I wouldn't drink the tap water in any African country. I've never found a place that doesn't have an abundant supply of bottled water available for cheap.
How can you know that a given bottle of water isn't just from that city's tap? Frankly, there are many big bottled water vendors in the US that do precisely that.
You can't know, so taking the cap off yourself is no guarantee that the water isn't contaminated. Probably the safest thing to drink anywhere is bottled beer given the prevalence of pasteurization for shelf life. I notice in cheap cafes Hanoi that they would hand me the opener with my bottle of Ha Noi Bia. Presumably so I could be assured that it was the real thing and hadn't been watered down.
Oh, and let's hear it for Pepto Bismol prophylaxis against amoebic dysentery.