Is the drive spinning up each time?

I used to have that problem when ripping, but Audiograbber has a feature that's called spin up disc before reading. It's under the rip offset tab under settings.

Don't know if that's your problem or not but maybe