I think the watch face was the culprit. I'm using one called Weather Watchface, and it just had an update today that listed one of the improvements as addressing a battery drain issue. I think this was independent of the 5.0 improvements which came in a previous update to the app.

I've disabled the fitness notifications too, but I can see that, for example, Moto Fit is still counting my steps. That's really annoying. I have to imagine that it's affecting battery life, but I'm more optimistic now. At one point today it took about an hour for the battery to drop 3%.