A reddit user recommended calibrating my IMU (despite it showing as normal) when they saw my avata jerkinng and flipping over in manual, and it's made a world of difference. If yours is doing something similar, might give the calibration a try.
I hope that they will release a patch firmware to fix such sudden misbehavior when using manual mode. A lot of avata users are also complaining and experiencing such misbehavior when yawing their drones causing big crashes.