Please select Into the mobile phone version | Continue to access the computer ver.
[DJI Certified] Intro to Virtual Stick: Beyond Waypoints
4602 16 2020-5-13
Uploading and Loding Picture ...(0/1)
o(^-^)o
dronelink0
lvl.3
United States
Offline

If you’ve been using the Mobile SDK for anything beyond FPV and manual flight, chances are you take advantage of the Mission Control APIs (iOS | Android). Mission Control gives developers an easy and reliable way to automate the flight and payload functions of the drone. The heart of Mission Control is the Waypoint Operator (iOS | Android), and these operators can even be stitched together via Timelines (iOS | Android) to solve multiple use cases from mapping to cinematography. But what happens when you reach the limit of these APIs (number of waypoints, types of waypoint actions, etc)?

The Mobile SDK has an answer: Virtual Stick (iOS | Android). In short, Virtual Stick allows your app to take control of the “virtual sticks” of the remote controller, essentially enabling your app to become the pilot, sending pitch, roll, yaw, and throttle (z-axis) commands to the drone, just like a human would.

Virtual Stick has actually been around since v1 of the Mobile SDK (November 2014) and was once the only way to automate the drone. When I started developing Autopilot in December 2014, none of the Mission Control APIs existed, so I had to create my own flight controller based on Virtual Stick. Back then the state-of-the-art in connectivity was the Phantom 2 Vision+, which was based on WiFi and suffered from low bandwidth high latency. Despite the challenges, it was possible to produce a minimum viable product with functionality like Follow and Focus modes. In the last six years, DJI has made some amazing advances with technologies like Lightbridge and OcuSync, which drastically increase the range and reduce the latency of the uplink and downlink of the remote controller, making the Phantom 2 Vision+ look primitive by comparison.

Lightbridge and OcuSync have opened the door to a whole new level of ground-based flight control that can take full advantage of the power and connectivity of mobile devices. For applications that require precision control (particularly in the z-axis) and adaptive flight based on machine learning at the edge, using Virtual Stick over these interfaces is the only option.

At first glance, Virtual Stick can seem overwhelming, and you might be wondering how to even get started. The short answer is: invoke setVirtualStickModeEnabled (iOS | Android). This works well in theory, but reality is always more complicated with plenty of edge cases:

  • What if the drone isn’t flying yet?
  • What if the remote controller isn’t in the right mode?
  • What if the drone loses GPS?
  • What if the remote controller gets disconnected?
  • What happens if the user backgrounds the app on iOS?


Answer: you need a robust and fault tolerant state machine that treats these edge cases like the rule rather than the exception. For example, here is some basic pseudo-code that shows how to takeoff and enable Virtual Stick:

enum State = {
    TakeoffStart,
    TakeoffAttempting,
    TakeoffComplete,
    VirtualStickStart,
    VirtualStickAttempting,
    VirtualStickComplete
}

virtualStickAttempts = 0
state = TakeoffStart

while true {
    switch state {
        case TakeoffStart:
            if flightController.isFlying {
                //skip the takeoff command if the drone is already flying
                state = TakeoffComplete
            }
            else {
                state = TakeoffAttempting
                //issue the takeoff command
                flightController.takeoff {
                    if error {
                        state = Deactivated
                    }
                    else {
                        state = TakeoffComplete
                    }
                }
            }
            break

        case TakeoffAttempting:
            //wait while attempting the takeoff command
            break

        case TakeoffComplete:
            //even though the takeoff command can succeed right away, that doesn't mean that
            //the takeoff is actually finished, so check flight controller state to be sure
            if flightController.isFlying && flightController.flightMode != AutoTakeoff {
                state = VirtualStickStart
            }
            break

        case VirtualStickStart:
            state = VirtualStickAttempting
            //issue the command to enable virtual stick
            flightController.setVirtualStickModeEnabled {
                if error {
                    //if it fails, retry it a few times
                    virtualStickAttempts++
                    if virtualStickAttempts > 3 {
                        state = Deactivated
                    }
                }
                else {
                    state = VirtualStickComplete
                }
            }
            break

        case VirtualStickAttempting:
            //wait while attempting to enable virtual stick
            break

        case VirtualStickComplete:
            //once virtual stick is enabled, the flight mode will change to Joystick,
            //meaning you can start sending virtual stick commands
            if flightController.flightMode == Joystick {
                //your special sauce to calculate commands!
                flightController.sendVirtualStickFlightControlData(commands)

            }
            else {
                //if the flight mode is no longer Joystick, it means something has changed
                //(RTH, GPS signal loss, remote controller flight mode switched, etc)
                state = Deactivated
            }
            break

        case Deactivated:
            //perform operations that give control back to the operator in a nice way
            //like stopping camera capture and resetting the gimbal to straight ahead
            camera.reset()
            gimbal.reset()
            return
    }

    //wait for the previous commands to reach the drone and for new telemetry to become available
    sleep 50ms
}

As you can see, handling even the basic cases (flying vs not flying) is non-trivial. Other cases (such as backgrounding the app on iOS) require even more consideration, especially if you are looking to enable a consistent experience across both iOS and Android devices. While the barrier to entry is definitely higher than the Mission Control APIs, the effort will be well worth it in the end.

Once you are confident in your state machine and have control of the drone, the next step is to actually send Virtual Stick commands. If you are creating an application for environments where GPS and compass signals are reliable, you will need to synthesize the telemetry coming from the flight controller state callbacks and perform the kinematic calculations fast enough to not delay the flight control loop. If you are operating in a GPS-denied environment or places with electromagnetic interference (compass-denied), you will need to use image recognition / optical flow / machine learning to compute the next set of commands based on the live video feed -- just like a human would do.

Curious to see just how far you can take a Virtual Stick application? I highly recommend that you check out my latest project at www.dronelink.com.

      

Drop me a line if you have any questions or are interested in taking advantage of the Dronelink SDK (think of it like a high-level abstraction layer for Virtual Stick). We even have a low-code way to the Dronelink SDK via Javascript on the web! Much of the SDK is actually published as open-source on our GitHub account, so you can actually see a working example of the above pseudo-code for iOS and Android.

Jim McAndrew
Founder & CEO
dev@dronelink.com
2020-5-13
Use props
Montfrooij
Captain
Flight distance : 2133671 ft
Netherlands
Online

Wow, this is new!
Nice to see!
2020-5-13
Use props
Woe
Captain
Flight distance : 4129268 ft
  • >>>
United States
Offline

Wow!!! Sounds cool.
2020-5-13
Use props
Montfrooij
Captain
Flight distance : 2133671 ft
Netherlands
Online

Woe Posted at 5-13 11:30
Wow!!! Sounds cool.

The first 'sponsored' thread I have ever seen on this forum
2020-5-13
Use props
Woe
Captain
Flight distance : 4129268 ft
  • >>>
United States
Offline

Montfrooij Posted at 5-13 12:25
The first 'sponsored' thread I have ever seen on this forum

A first for me as well.
2020-5-13
Use props
内涵阿哥
lvl.2
Flight distance : 791867 ft
  • >>>
China
Offline

Wow,   cool,   Many thanks to you guys!!!
2020-5-13
Use props
hdrpano
Second Officer
Flight distance : 452943 ft
  • >>>
Switzerland
Offline

Thanks for sharing!
I use a similar approach to create real 3D missions.

Steingrube 3D 1.JPG

2020-5-13
Use props
Montfrooij
Captain
Flight distance : 2133671 ft
Netherlands
Online

Woe Posted at 5-13 13:42
A first for me as well.

Not sure what experiment this is
2020-5-13
Use props
Woe
Captain
Flight distance : 4129268 ft
  • >>>
United States
Offline

Montfrooij Posted at 5-13 22:29
Not sure what experiment this is

I'm not sure either, but I'm interested.
2020-5-14
Use props
Montfrooij
Captain
Flight distance : 2133671 ft
Netherlands
Online

Woe Posted at 5-14 05:31
I'm not sure either, but I'm interested.

Sponsoring sounds good. Although it will probably be for DJI
2020-5-14
Use props
djiuser_OR8EW5uz3gSA
Second Officer

Canada
Offline

Fantastic article! I recently started looking into using Virtual Stick as I have found Timeline missions, while extremely convenient to code, are quite slow to complete.

However, I've noticed that when using a Mavic Pro with the DJISimulator (via the wireless SDK bridge) I cannot control the Virtual Sticks. Takeoff and Vertical Throttle work fine but I cannot pitch, yaw or roll regardless of the coordinate system used. Is this a known issue with the Mavic Pro? If so is there any workaround?
2020-6-9
Use props
dronelink0
lvl.3
United States
Offline

I would recommend not using the wireless bridge. Xcode has wireless debugging built in now and it is much more reliable.
2020-6-9
Use props
djiuser_OR8EW5uz3gSA
Second Officer

Canada
Offline

dronelink0 Posted at 6-9 10:00
I would recommend not using the wireless bridge. Xcode has wireless debugging built in now and it is much more reliable.

Thank you, that worked like a charm!
2020-6-10
Use props
lee82gx
Second Officer
Flight distance : 435049 ft
Malaysia
Offline

There is feedback from mavicpilots forum that the curved and rotating along a POI flight path results in jerky movements, is this normal?
2020-8-1
Use props
dronelink0
lvl.3
United States
Offline

Flight performance depends on many factors including hardware (how big and stable the drone is), environment (winds aloft, etc), device performance (which can be affected by battery level, ambient temperature, and direct sunlight), and the specifics of the mission plan (how fast and how tight are the curves). If you use a small drone in high wind with a low performance device that is low on battery and in direct sunlight on a hot day, you cannot expect the same performance as a large drone on a calm day with a brand new iPad Pro in 70 degree weather. When in doubt, try to mitigate as many of these factors as you can, the easiest of which are the flight path and speed (round out the corners and slow it down). In the end there is a 200ms round-trip latency with virtual stick (on average), meaning onboard code will always have a leg up relative to smoothness, but we have been able to achieve great results with many different drones and devices in varied conditions.
2020-8-1
Use props
djiuser_OR8EW5uz3gSA
Second Officer

Canada
Offline

Looking at your psudeo code you are checking if the flightMode is set to joystick mode before sending any virtual stick commands. I've noticed (at least with the Mavic Pro) the flight mode never changes to DJIFlightModeJoystick, it always stays as DJIFlightModeGPSAtti. Have you ever found this to be an issue on certain drone models? I guess calling isVirtualStickControlModeAvailable on the flight controller could be used as a workaround?
2020-9-17
Use props
cprest1n
lvl.2
United States
Offline

Hi thank you for this. I have what I think is a very basic question. You write:


The Mobile SDK has an answer: Virtual Stick (iOS | Android). In short, Virtual Stick allows your app to take control of the “virtual sticks” of the remote controller, essentially enabling your app to become the pilot, sending pitch, roll, yaw, and throttle (z-axis) commands to the drone, just like a human would.


But the human mode on the RC does not send pitch/roll/yaw, it sends yaw and up/down on the left stick and then cardinal move directions on the right stick. In other words humans are used to sending X, Y, Z axis commands  as well as heading.


This seems to me a big difference, the v-stick input set only controls pitch/roll/yaw, and using pitch+throttle to effect an XY-plane move is a big difference from what a human is used to. Using roll and throttle is more of thinking like an aviator than thinking like a human drone pilot.

So am I wrong about this? Is there a way to send commands just like a human using the sticks in mode 2?


4-23 12:25
Use props
Advanced
You need to log in before you can reply Login | Register now

Credit Rules