Depends on the piece of music. Yes, the drummer is usually responsible for keeping the tempo and is the anchor for the band, but sometimes a different instrument starts off the song or sets the pace of a given section. I seem to recall seeing the drummer for U2 talking about following the tempo of the echo effect on Edge's guitar, for example.