When flying my phantom without GPS, my Phantom is always flying to the left site. This is very annoying while flying indoor.  

Is there any setting where I can set these?

Have you tried doing a complete re-calibration of the IMU and seeing if the problem persists??  That would be a good place to start if you have not already done that.

It might also be wise to recalibrate the RC sticks. Give it a try and see if if helps correct this.

RedHotPoker

Just had the same thing happen to me. I called dji tech support yesterday and they said I had to send it in to their service department. They immediately emailed me an RMA number and the list of things, packing, labeling etc that I had to do. I went the next morning and sent everything back to them. Later in the day I received a shipping label for UPS for me to ship back. This was not told to me by the service tech so now I am out the shipping fees.

Hello MaxiKenter, IMU calibration and RC calibration will be necessary. Does the drone behave the same when it flies outside with ATTI mode?

RedHotPoker Posted at 2016-7-19 11:46
It might also be wise to recalibrate the RC sticks. Give it a try and see if if helps correct this.
...

Well, he implied it's normal when GPS is on, so I don't it has anything to do with RC sticks.

Covering the basic bases, trying to remove the simplest fixes first. Takes less than five minutes to do a stick cal. Ha

Offering free advice, is worth what I charge for it. ;-) or do like I,  buy a trouble free drone, then worry less.

RedHotPoker

This problem was recently posted by someone else.  After uploading their info to http://healthydrones.com/ they found that they were actually leaning a bit to the side on the control stick, despite the fact that they thought they weren't.  Stick calibration can also cause this.  Be sure everything is calibrated and then try it.  Upload the log to healthydrones and see what it says.  Personally, I find the sticks could be better about locking into the center detents and I cause mine to drift a little also.