Instead of doing this all as a one-liner I just split it off and added a bit of logic. Now I only call perl if grep finds matching files.

This solve the problem. Finally. I probably should have just attacked this from the script to begin with instead of spending all the initial time testing from the command line.