Not necessarily, it could just as well reside where the data is located. It could also be re-built (or re-checked) every time you reboot your computer, which could cause these sorts of slowdowns. Maybe you can define this in in Calibre's preferences?