You'll have issues when using the nVidia binary drivers if you're a kernel developer or if you ever want to submit a kernel bug report. It'll taint the kernel and they'll just throw out your bug report since in theory the nv driver could have been stomping all over your memory and you wouldn't know.
Also aren't the binary drivers based on the Windows code? It's how they can release new versions so quickly after the Windows driver comes out. They have some abstraction layer built into it that allows the Windows driver code be wedged into the Linux kernel. This is all what I've heard anyway so it could be false.

- Trevor