Agreed, pretty much any inexpensive handheld will give you enough functionality to find geocaches. All you really need is the compass and the distance to the waypoint. Everything beyond that is gravy.

When I started, I just picked up a Magellan Meridian for $100 on ebay to get started. Works great. Granted, I probably have to circle and hunt for a bit longer because it's not quite as accurate as some, but I've never had trouble getting to the caches. Perhaps one of these days I'll upgrade to something a little nicer, but for now it does the job.