Please select Into the mobile phone version | Continue to access the computer ver.
[DJI Certified] Intro to Virtual Stick: Beyond Waypoints
321 9 5-13 10:15
Uploading and Loding Picture ...(0/1)
o(^-^)o
dronelink0
lvl.2
United States
Offline

If you’ve been using the Mobile SDK for anything beyond FPV and manual flight, chances are you take advantage of the Mission Control APIs (iOS | Android). Mission Control gives developers an easy and reliable way to automate the flight and payload functions of the drone. The heart of Mission Control is the Waypoint Operator (iOS | Android), and these operators can even be stitched together via Timelines (iOS | Android) to solve multiple use cases from mapping to cinematography. But what happens when you reach the limit of these APIs (number of waypoints, types of waypoint actions, etc)?

The Mobile SDK has an answer: Virtual Stick (iOS | Android). In short, Virtual Stick allows your app to take control of the “virtual sticks” of the remote controller, essentially enabling your app to become the pilot, sending pitch, roll, yaw, and throttle (z-axis) commands to the drone, just like a human would.

Virtual Stick has actually been around since v1 of the Mobile SDK (November 2014) and was once the only way to automate the drone. When I started developing Autopilot in December 2014, none of the Mission Control APIs existed, so I had to create my own flight controller based on Virtual Stick. Back then the state-of-the-art in connectivity was the Phantom 2 Vision+, which was based on WiFi and suffered from low bandwidth high latency. Despite the challenges, it was possible to produce a minimum viable product with functionality like Follow and Focus modes. In the last six years, DJI has made some amazing advances with technologies like Lightbridge and OcuSync, which drastically increase the range and reduce the latency of the uplink and downlink of the remote controller, making the Phantom 2 Vision+ look primitive by comparison.

Lightbridge and OcuSync have opened the door to a whole new level of ground-based flight control that can take full advantage of the power and connectivity of mobile devices. For applications that require precision control (particularly in the z-axis) and adaptive flight based on machine learning at the edge, using Virtual Stick over these interfaces is the only option.

At first glance, Virtual Stick can seem overwhelming, and you might be wondering how to even get started. The short answer is: invoke setVirtualStickModeEnabled (iOS | Android). This works well in theory, but reality is always more complicated with plenty of edge cases:

  • What if the drone isn’t flying yet?
  • What if the remote controller isn’t in the right mode?
  • What if the drone loses GPS?
  • What if the remote controller gets disconnected?
  • What happens if the user backgrounds the app on iOS?


Answer: you need a robust and fault tolerant state machine that treats these edge cases like the rule rather than the exception. For example, here is some basic pseudo-code that shows how to takeoff and enable Virtual Stick:

enum State = {
    TakeoffStart,
    TakeoffAttempting,
    TakeoffComplete,
    VirtualStickStart,
    VirtualStickAttempting,
    VirtualStickComplete
}

virtualStickAttempts = 0
state = TakeoffStart

while true {
    switch state {
        case TakeoffStart:
            if flightController.isFlying {
                //skip the takeoff command if the drone is already flying
                state = TakeoffComplete
            }
            else {
                state = TakeoffAttempting
                //issue the takeoff command
                flightController.takeoff {
                    if error {
                        state = Deactivated
                    }
                    else {
                        state = TakeoffComplete
                    }
                }
            }
            break

        case TakeoffAttempting:
            //wait while attempting the takeoff command
            break

        case TakeoffComplete:
            //even though the takeoff command can succeed right away, that doesn't mean that
            //the takeoff is actually finished, so check flight controller state to be sure
            if flightController.isFlying && flightController.flightMode != AutoTakeoff {
                state = VirtualStickStart
            }
            break

        case VirtualStickStart:
            state = VirtualStickAttempting
            //issue the command to enable virtual stick
            flightController.setVirtualStickModeEnabled {
                if error {
                    //if it fails, retry it a few times
                    virtualStickAttempts++
                    if virtualStickAttempts > 3 {
                        state = Deactivated
                    }
                }
                else {
                    state = VirtualStickComplete
                }
            }
            break

        case VirtualStickAttempting:
            //wait while attempting to enable virtual stick
            break

        case VirtualStickComplete:
            //once virtual stick is enabled, the flight mode will change to Joystick,
            //meaning you can start sending virtual stick commands
            if flightController.flightMode == Joystick {
                //your special sauce to calculate commands!
                flightController.sendVirtualStickFlightControlData(commands)

            }
            else {
                //if the flight mode is no longer Joystick, it means something has changed
                //(RTH, GPS signal loss, remote controller flight mode switched, etc)
                state = Deactivated
            }
            break

        case Deactivated:
            //perform operations that give control back to the operator in a nice way
            //like stopping camera capture and resetting the gimbal to straight ahead
            camera.reset()
            gimbal.reset()
            return
    }

    //wait for the previous commands to reach the drone and for new telemetry to become available
    sleep 50ms
}

As you can see, handling even the basic cases (flying vs not flying) is non-trivial. Other cases (such as backgrounding the app on iOS) require even more consideration, especially if you are looking to enable a consistent experience across both iOS and Android devices. While the barrier to entry is definitely higher than the Mission Control APIs, the effort will be well worth it in the end.

Once you are confident in your state machine and have control of the drone, the next step is to actually send Virtual Stick commands. If you are creating an application for environments where GPS and compass signals are reliable, you will need to synthesize the telemetry coming from the flight controller state callbacks and perform the kinematic calculations fast enough to not delay the flight control loop. If you are operating in a GPS-denied environment or places with electromagnetic interference (compass-denied), you will need to use image recognition / optical flow / machine learning to compute the next set of commands based on the live video feed -- just like a human would do.

Curious to see just how far you can take a Virtual Stick application? I highly recommend that you check out my latest project at www.dronelink.com.

      

Drop me a line if you have any questions or are interested in taking advantage of the Dronelink SDK (think of it like a high-level abstraction layer for Virtual Stick). We even have a low-code way to the Dronelink SDK via Javascript on the web! Much of the SDK is actually published as open-source on our GitHub account, so you can actually see a working example of the above pseudo-code for iOS and Android.

Jim McAndrew
Founder & CEO
dev@dronelink.com
5-13 10:15
Use props
Montfrooij
Captain
Flight distance : 1859961 ft
Netherlands
Offline

Wow, this is new!
Nice to see!
5-13 10:24
Use props
Woe
Captain
Flight distance : 3897244 ft
  • >>>
United States
Offline

Wow!!! Sounds cool.
5-13 11:30
Use props
Montfrooij
Captain
Flight distance : 1859961 ft
Netherlands
Offline

Woe Posted at 5-13 11:30
Wow!!! Sounds cool.

The first 'sponsored' thread I have ever seen on this forum
5-13 12:25
Use props
Woe
Captain
Flight distance : 3897244 ft
  • >>>
United States
Offline

Montfrooij Posted at 5-13 12:25
The first 'sponsored' thread I have ever seen on this forum

A first for me as well.
5-13 13:42
Use props
内涵阿哥
lvl.2
Flight distance : 791867 ft
  • >>>
China
Offline

Wow,   cool,   Many thanks to you guys!!!
5-13 19:00
Use props
hdrpano
lvl.2
Flight distance : 442185 ft
  • >>>
Switzerland
Offline

Thanks for sharing!
I use a similar approach to create real 3D missions.

Steingrube 3D 1.JPG

5-13 20:58
Use props
Montfrooij
Captain
Flight distance : 1859961 ft
Netherlands
Offline

Woe Posted at 5-13 13:42
A first for me as well.

Not sure what experiment this is
5-13 22:29
Use props
Woe
Captain
Flight distance : 3897244 ft
  • >>>
United States
Offline

Montfrooij Posted at 5-13 22:29
Not sure what experiment this is

I'm not sure either, but I'm interested.
5-14 05:31
Use props
Montfrooij
Captain
Flight distance : 1859961 ft
Netherlands
Offline

Woe Posted at 5-14 05:31
I'm not sure either, but I'm interested.

Sponsoring sounds good. Although it will probably be for DJI
5-14 22:34
Use props
Advanced
You need to log in before you can reply Login | Register now

Credit Rules