The image below is a single frame captured directly from the S1 video stream from external code without going through the Robomaster app at all! It seems there is some tweaking to do (look at all the banding on the image) but this basically mean I can now capture the video stream and feed it to something like OpenCV running locally on the controlling machine.

The code is basically hacked up now to save the single frame (code is in my repository) but if this works, everything else will related to video capture will.

Robomaster S1 camera feed capture

This is fantastic! Thank you!

Thank you BGA, time to take over the world

WoW Bravo, and excellent hack!

Curious to what the usable frame rate is?

MarkusXL Posted at 12-19 16:46
WoW Bravo, and excellent hack!

Curious to what the usable frame rate is?

The S1 transmits at 30 FPS so that is what I got. I did an initial attempt to hook it up to OpenCV but failed (I am pretty sure it is mostly because I have to run this on Windows due to the dependency on the unitybridge.dll file).

I am considering an IPC approach where I isolate the windows code by wrapping it around an IPC server that I can run with wine. Then I start it from the actual Linux code and this minimizes the windows dependency.

The interesting thing is that here the unitybridge.dll is both a blessing and a curse. A blessing because it decodes the video stream and what I get is actually fully decoded RGB24 frames. No need to do any decoding. And a curse because the same thing results in each frame being around 2.7 MB. at 30 fps this is 81 mb per second which is high enough to make me worry about the IPC implementation. We will see.

Sounds like you're getting uncompressed frames.   I did a quick calculation: a frame being 1440x1080 (assuming 1080p 4:3 aspect ratio) with 3 bytes per pixel (8 bits per RGB) comes out to 4.6MB.  With 720p I calculate 2MB.   With 720p at 16:9 I get 2.7MB.   Then again you did say unencoded, which means raw frames.

DanMan32 Posted at 1-6 08:19
Sounds like you're getting uncompressed frames.   I did a quick calculation: a frame being 1440x1080 (assuming 1080p 4:3 aspect ratio) with 3 bytes per pixel (8 bits per RGB) comes out to 4.6MB.  With 720p I calculate 2MB.   With 720p at 16:9 I get 2.7MB.   Then again you did say unencoded, which means raw frames.

Not sure I understand the point you are trying to make but, in any case (and as I mentioned in one of my replies), unitybridge does the decoding and I get decoded frames. I also showed the math in my reply (it is 2.7 Mb per frame @720p or 81 mb/s). Note this is only relevant because I am uising an IPC solution between Wine and a native linux app to be able to do Linux development. The actual S1 stream is H2664 and is a lot smaller than what you get with unencoded frames, of course.

I know many worry about video quality based on bitrate.   I didn't catch at first that by "decoded", you meant full video uncompressed, which indeed is 2.7MB when video is 1280x720.  I would think that a good thing, unless transmitting it over slow channels that can't handle the rate.

Awesome capture.

Lucas775 Posted at 2-8 14:25
Awesome capture.

Thanks. You can also find actual videos in other posts.

BGA Posted at 2-9 18:11
Thanks. You can also find actual videos in other posts.

I will sure look for them.