Hey all.

I've been finding a lot of questions about the release date of the new firmware and why it is taking so long and coming from an embedded design background I thought that it would be a worthwhile exercise posting a little explanation about what goes into a firmware release and how long one can take. Hopefully it will help those who are anxious to empathise a bit more with DJI and be more understanding.

There are generally two types of release; maintenance and feature addition. Sometimes a release will contain a blend of both.

A feature release is normally the simpler of the two. It's built on top of what is assumed to be bug free code and the evaluation consists of just ensuring that the new additions function correctly and don't break any of the old stuff.

A release that is primarily for maintenance will involve fixing bugs. As it happens these releases can often take much longer to get out of the door than a feature addition release. The reason is basically down to the time that it takes to identify the root cause of the bugs. It's important to remember that each release is tested very thoroughly for weeks if not months before it is made and if bugs still creep through then they (or at least the root cause of them) must be particularly difficult to find. Normally in a Web or desktop development environment the developer can control the flow of code by viewing its execution, step by step, in real time and setting up "breakpoints" that allow him to pause the code when it reacts to certain inputs. Using this technique he can go through a range of inputs until the code produces the buggy behavior and when it does he can watch exactly what it is doing so that the root cause becomes obvious. This is not the case with DJI.

Unfortunately their product cannot just be set up on the bench and expected to replicate the bug. They have to fly it untethered which means that they can't see what the code is doing when the bug happens, they can only see the cause and effect and from there on they have to guess. Narrowing down the root cause of a bug like this is normally an iterative process and can take weeks if not months. Thereafter, final QA testing is still required and if there is a problem found during the testing then it's back to the drawing board.

No doubt DJI are eager to release a quality piece of firmware after the heat from 1.2.6/8 so I'm sure that they have repeated this cycle a few times by now. That is why they are struggling to give a date. They just can't release with the knowledge that there are flight critical bugs still present in the firmware.

The bright side of this is that every day that DJI hang on to the release is another day that they have to sniff out and fix such bugs. Rather we all wait as long as it takes than have DJI buckle under pressure and release early only to have more of our birds falling out of the sky.

I suppose if you really want the new firmware sooner (and have the stomach for it) then you could sign up as a beta tester and help them stress test the bugs out of it. Applicants with a foam padded wind tunnel with full GPS signal are preffered ;)

https://dji.centercode.com/signup/

Wake me up when a release is tested and released on the masses.

A far too sensible post ;-)

I suspect they can get a fair amount of debug data over light bridge and by writing to the SD Card. They should have not problems from that perspective. But I do admit it is a hell of a lot of variables from the environmental perspective but the firmware is 1 and 0s, it's not magic, its sensor input, calculation, control movement.

What is peculiar is the delay in releasing the previous firmware as a new version for those affected by the problems. It depends on the method they use to patch the firmware. The packages are either going to apply code changes to specific part of the firmware almost like a series of hot fixes to computer operating system, or it contains the entire firmware.

They may be having problems getting the previous version to run reliably over 1.2.6/8, perhaps they tried it for a couple of weeks and decided it was taking too long so opted to crack on with the next full release and fix the problems in that. It also suggests that DJI employees and associates who often post on here, other forums and their tech support are largely kept in the dark as to what their engineering departments are doing.

Your could almost hear their head of engineering saying "You told them what?".

They're also having to balance that with the needs of their SDK developers as many have waited weeks for DJI to get them approved via Apple and for an SDK version of the firmware, thankfully that now seems to be progressing.

young.selina@ro Posted at 2015-7-26 17:23
[/td][/tr]
[/table]

u make a lot of sence but I am so mad thatdji released the p3 before it was ready.i bought it for the gs and poi and the follow me.the aerosky x350 does all that with a twenty min fly time and a four hundred doller price tag.i really don't see what is taking so long these people developed three versions of ps this should be a drop in the bucket for them

yea man but these are geniouses that do this stuff with their eyes closed I think dji is purposely not releasing it.the p2 has the new stuff it shouldn't be that hard for these guys they just don't have incentives.im sueing them in small claimes because they sold a product that was supposed to come with poi and gs and follow me.the aerosky x350 does it all for four hundred bucks no waiting and a third of the cost.i would rather have a vision + then this over priced toy

robb getts Posted at 2015-7-26 22:31
yea man but these are geniouses that do this stuff with their eyes closed I think dji is purposely n ...

I have a P3, I do not think that it is a toy.  Please don't rant and complain. Go buy what you desire for a few hundred bucks.  

I guess it takes so long because it is the first time they really test a FW like it should be tested before releaseing it. Thats why i can wait now, to many times in the past the FW wasnt tested and they even pulled it back. The only thing that angers me a bit is that this is normally done before a product is released but i guess they dont had the time because other manufactors rushing out with their products. I dont think the Phantom is a toy but sometimes i feel i have bought a toy for them or i feel like i am a customer that spend to less money for beeing respected enough for them. For example the inspire customers are more respected, the gimbal system works much better, less quality issues but in the end its the same camera. In the cheaper toy grade range ppl have to accept jello effect and other quality control issues and thats why i feel sometimes like i have bought a toy. But in the end this is 1400 euro, the price of a full format DSLR. I just whish they raise the gimbal and shell quality to the 1400 Euro level. Yuneec manages to have a jello free gimbal, dont know why that is, maybe because of less vibrations from the larger shell or a better QC.

Great original post! Makes a ton of sense, BUT I wish they'd pour a little more steam on at DJI (go faster,that is).

Myself and many others have had a firmware update in version 1.1.9 that was rock solid and then there were others that were fine with 1.1.8 and then to be told you must update to 2.1.6 that in my case took me out of the game for weeks now.
So this is why a lot of folks are saying lets have the old working firmware versions with whatever name they want to call them, back up to download as a stop gap while we wait for Dji to test.
Because in the word of Bruce Lee they have offended my family

young.selina@ro Posted at 2015-7-26 17:23
[/td][/tr]
[/table]

There are a few interesting quotes about simulation on the internet which are a bit to graphic to post here...

The thing about simulation is that the input data is only as diverse as the creator of it can cook up. So yes, you can catch logical errors based on input data streams and stimuli that an engineer has designed. Unfortunately the real world throws variables at us that we would never have conceived in our wildest imagination and this is where the really interesting fun begins. How does the code respond to them? This can only be found out by real world testing.

A product that has been tested in a simulation environment will work very well as long as it operates within an environment that is similar to that. Perhaps this is in fact the approach that DJI have been taking which explains why they were able to get releases out in rapid succession but every now and then a drone falls out of the sky when it does not know how to respond to some random data or code execution that occurs.

For instance I have found times when a sensor simply locks up entirely and stops spitting out reasonable data. The only way to bring it back to life is a full power cycle. In other cases sensors may just spit out data that is normal...normal...normal...massive spike...normal...normal...normal... this too can cause the drone to throw a wobbly. Events like these can only be simulated once engineers know that they are possible which can only be discovered by watching a drone malfunction in the real world and then asking why. There are also CPU anomalies that can happen where RAM can become corrupt with voltage spikes or the execution of code jumps from one point to another in an unexpected manner. All of these can actually be handled but engineers need to anticipate them and then design around the worst case scenarios.Testing for these types of events is all a game of probabilities with more diligent testers reducing the time required to observe and record a failure.

I can't argue that their many years of experience in this field should stand them in good stead and allow them to produce quality firmware but I can say that they have grown tremendously which must surely be unsettling to an engineering department and to keep the in house experience being distributed to all the new engineers is no easy task. With the P3 being built based on the new inspire platform I am guessing that there were a lot of new hands on deck.

Regarding the comment on the beta testers, I would treat any info submitted from an outside source as questionable if I were DJI. It does not mean that it holds no value but it should be scrutinized. Instead I would ensure that as many competent company employees as possible know how to fly and are instructed to take a certain amount of time each day to make flights and record the data. This will provide the diversity and reliability of data.

sdharris Posted at 2015-7-26 18:40
I suspect they can get a fair amount of debug data over light bridge and by writing to the SD Card.  ...

Yea, I reckon that their releases are patch releases. You could probably tell if you look at the size of the binaries from release to release. If one ever gets smaller in a later release then you almost can confirm that it is a patch release.

The SDK dev point is also very valid. I think that they are struggling to meet the needs of all the departments that are tugging on them. I don't envy the position that they are in at all.

My P3 is still on 1.1.9 and I plan to keep it there for the remainder of the flying season which may be 2 more months.  After that it sits for winter until next spring....  I recently bought an Inspire, but returned it unused after deciding to wait and see how issues with that get resolved.  

Rob