My battery will charge until I get the full 4 LEDs, but it never turns off.  The other battery I have charges to 4 full LEDs and then turns off.  Is this a faulty battery?

It sounds like the final balancing is taking a long time, have you tried leaving it for several hours?
If it seems normal in use, running it down below 10% then recharging might fix it.
The seller would probably replace if you described your concerns.

Stupid question, but did you try to turn it off?

Could you please make a short video of this issue? When you put the battery into the aircraft, is there any error message shows up from DJI GO app?