Ok I scratched my lens on the inside and thought.. Hey I'll just buff that out a bit with my dremel.

The inside of the factory lenses are coated with an anti-RF coating, so buffing a spot on the inside of the lens would cause a "blank spot" that would look funny. You're right, you should have just left it.