That's because the Mac's "Delete" key is really backspace.

If you keep your files on a network share, you'll have them deleted immediately, even if you'd prefer a recycle bin.

I don't believe there's such a shortcut when using the Finder and I can't remember if there's one if using the Finder replacement (which I much prefer), Path Finder.