I've installed the recent app and firmware update (800), but I'm still not clear on how to make sure that all five of my extra batteries receive the updated firmware. Does the Mavic update the battery whenever the dead battery is swapped out and a fresh one put in? Or do I need to manually update each battery, and if so, how exactly do I do that?

You'll need to power up the aircraft  and remote controller while hooked up to a DJI Go app and have Wi-Fi access and then put each battery in Mavic. if you put a battery in it that has old firmware you will get a notice that says Miss Matched firmware upgrade now select yes do that with each battery you'll need to do this after the new 800 firmware update.

Also the two middle lights on the battery will be doing things so you'll know it's upgrading.

Good question and i might add that yes, you will be prompted to update when you swap out old battery for battery with older FW version.
Not sure at this point if you need wifi or any connection for this to be achieved.
A few weeks ago i updated battery FW in the field as above with no connections which made me suspicious about needing wifi.
I think the FW is delivered to the battery in need via the onboard FW version. Took a few minutes though and maybe wifi would be quicker, not sure.

I sure wish I could see evidence of my batteries (4) ever being upgraded... I never see the lights or the messages, they just always work... across 4 versions now. Your mileage may vary.

Thanks! Very helpful!

Interesting info about the batteries. Specifically he lights that indicate the update process. Got 4 batteries and have update the Mavic several times and not once did it flag up the FW on the batteries. My goggles haven't either yet...

I have two batteries. First one was updated with the drone itself (you see it because second light flashes still, third flashes intermitently and very fast) When I popped in the second battery I had a message in the app telling me that there was a firmware inconsistency, slide to upgrade. Some more led blinking on the battery and I was good to go!