I charged both my batteries two weeks ago and only used one of the battery a week later.  Now another week in, I took the other battery out of my bag and it is "warm".  

Did a quick search and from other DJI drone users, it's supposed to discharging due to non-usage for a certain period of time.  Anyone can chime in?  

Got worried since I didn't even touch the battery after I charged it.  Thought it was going to blow up.  

refer to manual, page 27 DJI Intelligent flight battery functions..

The battery will be discharged if you do not use it for 10 days. So the battery might get warm when it is discharging. Please power on the drone with this battery and connect it to DJI GO 4 app first to see if you can get any reminders. If no error messages, the battery is working alright. Should you have any further questions, please feel free to contact us. We'd glad to help.