Suppose we defined morality as conforming to a code of ethics that caused the least amount of harm to others?

John Stuart Mill would definitely agree with you there.

Oddly enough - when I searched for Mill's Harm Principle, this very related topic also came up in the top 10 results. Freaky.