I certainly don't remember turning it off, I wasn't aware that the setting was there.
My iPhone was setup from a restore, maybe Facetime gets set turned off in those cases ?
If you restore a non iPhone 4 backup to an iPhone 4, Facetime defaults to off for some reason. If you instead setup an iPhone 4 as a new device, it defaults to on. I'm not sure if this same behavior occurs on the newest iPod Touch.
This may have changed with factory units coming with something newer then 4.0, but that was the experience I saw when I set my iPhone up new, and a friend restored his with his 3G backup.