Yes I think you are going in correct direction because you can use lot of reusable things from existing framework. Like loggers,reports,base class with language specific helper methods / utility methods like (xls,csv,txt,json) file reader/writer etc
More over you can add driver implementation in existing classes and then write one class which can create wrapper over appium java client methods like click,sendkeys,swipe,zoom etc.
There may be configuration challenges like where you have to run only mobile test cases so you have to explore that area. I believe its a good way of creating and learning a robust framework for handling different devices specially test cases work for a single AUT. All the best.
