No, it's just connected to the RTC. Leaving it in there shouldn't cause any problems.

Hugo