Hi @jheideman1,
This issue where you see the driver "failing to swipe" is particular to Appium 1.6.+ and the Appium project is currently working on a solution that should be implemented in the next version of Appium (i.e 1.6.4).
To explain things in more detail:
With the recent changes in automation behavior on iOS 10+ tests, Appium had to find a workaround to automate the "swipe" action using the "drag" and "drop" methods. This solution, as you have experienced is causing the "swipe" action to fail in certain scenarios like the one you are describing here.
The XCTest framework has now provided a native "swipe" method that will allow the user to automate "swiping" as if an actual user was performing the swiping. We are currently working on automating this new "swipe" method in Appium 1.6.4.