If you’ve been using the Mobile SDK for anything beyond FPV and manual flight, chances are you take advantage of the Mission Control APIs ( iOS | Android). Mission Control gives developers an easy and reliable way to automate the flight and payload functions of the drone. The heart of Mission Control is the Waypoint Operator ( iOS | Android), and these operators can even be stitched together via Timelines ( iOS | Android) to solve multiple use cases from mapping to cinematography. But what happens when you reach the limit of these APIs (number of waypoints, types of waypoint actions, etc)?
The Mobile SDK has an answer: Virtual Stick ( iOS | Android). In short, Virtual Stick allows your app to take control of the “virtual sticks” of the remote controller, essentially enabling your app to become the pilot, sending pitch, roll, yaw, and throttle (z-axis) commands to the drone, just like a human would.
Virtual Stick has actually been around since v1 of the Mobile SDK (November 2014) and was once the only way to automate the drone. When I started developing Autopilot in December 2014, none of the Mission Control APIs existed, so I had to create my own flight controller based on Virtual Stick. Back then the state-of-the-art in connectivity was the Phantom 2 Vision+, which was based on WiFi and suffered from low bandwidth high latency. Despite the challenges, it was possible to produce a minimum viable product with functionality like Follow and Focus modes. In the last six years, DJI has made some amazing advances with technologies like Lightbridge and OcuSync, which drastically increase the range and reduce the latency of the uplink and downlink of the remote controller, making the Phantom 2 Vision+ look primitive by comparison.
Lightbridge and OcuSync have opened the door to a whole new level of ground-based flight control that can take full advantage of the power and connectivity of mobile devices. For applications that require precision control (particularly in the z-axis) and adaptive flight based on machine learning at the edge, using Virtual Stick over these interfaces is the only option.
At first glance, Virtual Stick can seem overwhelming, and you might be wondering how to even get started. The short answer is: invoke setVirtualStickModeEnabled ( iOS | Android). This works well in theory, but reality is always more complicated with plenty of edge cases:
What if the drone isn’t flying yet? What if the remote controller isn’t in the right mode? What if the drone loses GPS? What if the remote controller gets disconnected? What happens if the user backgrounds the app on iOS?
Answer: you need a robust and fault tolerant state machine that treats these edge cases like the rule rather than the exception. For example, here is some basic pseudo-code that shows how to takeoff and enable Virtual Stick:
enum State = { TakeoffStart, TakeoffAttempting, TakeoffComplete, VirtualStickStart, VirtualStickAttempting, VirtualStickComplete }
virtualStickAttempts = 0 state = TakeoffStart
while true { switch state { case TakeoffStart: if flightController.isFlying { //skip the takeoff command if the drone is already flying state = TakeoffComplete } else { state = TakeoffAttempting //issue the takeoff command flightController.takeoff { if error { state = Deactivated } else { state = TakeoffComplete } } } break
case TakeoffAttempting: //wait while attempting the takeoff command break
case TakeoffComplete: //even though the takeoff command can succeed right away, that doesn't mean that //the takeoff is actually finished, so check flight controller state to be sure if flightController.isFlying && flightController.flightMode != AutoTakeoff { state = VirtualStickStart } break
case VirtualStickStart: state = VirtualStickAttempting //issue the command to enable virtual stick flightController.setVirtualStickModeEnabled { if error { //if it fails, retry it a few times virtualStickAttempts++ if virtualStickAttempts > 3 { state = Deactivated } } else { state = VirtualStickComplete } } break
case VirtualStickAttempting: //wait while attempting to enable virtual stick break
case VirtualStickComplete: //once virtual stick is enabled, the flight mode will change to Joystick, //meaning you can start sending virtual stick commands if flightController.flightMode == Joystick { //your special sauce to calculate commands! flightController.sendVirtualStickFlightControlData(commands)
} else { //if the flight mode is no longer Joystick, it means something has changed //(RTH, GPS signal loss, remote controller flight mode switched, etc) state = Deactivated } break
case Deactivated: //perform operations that give control back to the operator in a nice way //like stopping camera capture and resetting the gimbal to straight ahead camera.reset() gimbal.reset() return }
//wait for the previous commands to reach the drone and for new telemetry to become available sleep 50ms }
As you can see, handling even the basic cases (flying vs not flying) is non-trivial. Other cases (such as backgrounding the app on iOS) require even more consideration, especially if you are looking to enable a consistent experience across both iOS and Android devices. While the barrier to entry is definitely higher than the Mission Control APIs, the effort will be well worth it in the end.
Once you are confident in your state machine and have control of the drone, the next step is to actually send Virtual Stick commands. If you are creating an application for environments where GPS and compass signals are reliable, you will need to synthesize the telemetry coming from the flight controller state callbacks and perform the kinematic calculations fast enough to not delay the flight control loop. If you are operating in a GPS-denied environment or places with electromagnetic interference (compass-denied), you will need to use image recognition / optical flow / machine learning to compute the next set of commands based on the live video feed -- just like a human would do.
Curious to see just how far you can take a Virtual Stick application? I highly recommend that you check out my latest project at www.dronelink.com.
Drop me a line if you have any questions or are interested in taking advantage of the Dronelink SDK (think of it like a high-level abstraction layer for Virtual Stick). We even have a low-code way to the Dronelink SDK via Javascript on the web! Much of the SDK is actually published as open-source on our GitHub account, so you can actually see a working example of the above pseudo-code for iOS and Android.
Jim McAndrew
Founder & CEO
dev@dronelink.com
| 
lvl.2
Captain
lvl.4
lvl.1
