I am new to drones, and the spark is my first one. When I fly during the day, it is okay, flies as expected, but at night, it drifts around on its own. Is it really like this?

Yep, it's really like this.

During the day, it's able to use its downward sensors to help hold its horizontal position. At night, those sensors don't have enough light to "see" the ground. Without the sensors, it's only able to use GPS (which is not 100% accurate).

Yes, when I fly at night (which is rare) I keep it close.

Be really careful if you want to fly at night!

BTW welcome to the forum mate!

Hi there, sorry for the trouble. We do advise you to be much more careful when you operate the drone at night, even though there is light. At night, the vision system may not work normally. We highly recommended flying your drone in the daytime for us to avoid an incident. If you are new to fly the drone, we also advise you to fly the drone within the range of visibility in the open area without obstacle and signal interference. If you have other concern, please don't hesitate to contact us. Thanks for your support.

How shallow to the ground are you flying, please?

I flew it at around just 3 meters when I noticed it start to drift.

That could be a vps, which senses ground, but because of no(t enough) light, just a "black carpet".
Any wind, fan blowing?
Without enough light and/or above a very homogen surface, the Spark is "irritated"
Even over some carpet, steady hovering, you pull the carpet slowly, it will follow
There is a video out about this, somewhere.

I once used my Spark indoor and the light was not strong enough in the room, so the Spark went flying side way very fast until it hits the sofa and fell on it. I did not even had time to react.  Good thing it was just a soft chair. Now when I fly it inside I make sure there is enough light and it stay very stable. I guess outside it is the same problem and it will drift.