I know exactly what you're talking about. But thanks for the message anyway, just in case someone else tried it without that extra care. I'll be using my drill press and will have the depth locked. The fascia will be clamped to the press.
I would consider modifying only the bolts to make them fit better, but it would require grinding off too much of the head. So much that it might not leave very much depth for the hex driver afterwards. So in the end there will be a bit of both. A bit drilled out of the hole and a bit ground off the bolt.
Bruno
/edit/
Here's an image of the semi-rough application of the epoxy.