I’m new to this, I just got my Mavic pro and goggles and it looks to me like you could Fly it without opening the DJI GO app. Would there be any upside or downside to doing this?

Hi, there. Please connect your Mavic with app first and then you can link the mavic pro with goggles to fly your drone. Or you will get the limilt distance since you do not log in DJI account. Thanks for your support.

The Goggles integrate very well with the Mavic Pro, you can even control Goggles menus using Mavic Pro RC.
As DJI Elektra pointed out the downside, if it is at all, is that the aircraft still needs to get confirmation of account sign on from DJI Go 4 App'. If it does not see this on display device then there is a 30m height and 50m radial limit on flying area.
Other than having to have a device connected to RC there should be no problems. You can log DJI Go 4 into your account at home, or wherever wifi available, and that's it, as long as you don't log off again you can close and reopen DJI Go 4 and it will appear 'logged in', even if no wifi nor cell signal available where flying.

Edit:- One nice advantage is that you can continue to fly using display and let somebody else have a birdseye view using goggles (which have Ocusync wireless link to aircraft).
Then when you are using Goggles they can be your 'Observer', keeping eyes and ears open for any other aircraft or obstacles where you're flying FPV (First Person View).

Great info, thanks loads