Is it possible to fly forward when the mavic mini gimbal is adjusted to point upwards to its limit? As soon as I flew it moving forward, the gimbal dropped back down to horizontal position. When static hovering it's fine only when going forward. Anyone else found this?

Only on cinema mode this will not happend. This is option to save gimbal motors

You may want to look at your flight settings. In the Mavic Mini User Manual, on page 23, in the paragraph "Gimbal Operation Modes" is a description of the differences between the Follow Mode and the FPV (First Person View) Mode. From the manual:
Follow Mode: the angle between the gimbal's orientation and aircraft front remains constant at all times.
FPV Mode: The gimbal synchronizes with the movement of the aircraft to provide a first-person flying experience.

Personal note: I fly in follow mode so that I can see what is going on below me.  I am training myself to use the aircraft as a SAR aid.