I wouldn't drink the tap water in any African country. I've never found a place that doesn't have an abundant supply of bottled water available for cheap.
How can you know that a given bottle of water isn't just from that city's tap? Frankly, there are many big bottled water vendors in the US that do precisely that.
The big trick (at least in India) is to make sure that the bottle is
unopened when it's given to you. Don't be afraid to send an opened bottle back. Crack it yourself, or don't drink it. People aren't above refilling bottles.