Running out of RAM on Linux box?

I've got my email and web server hosted on a VPS. This VPS doesn't really have enough memory assigned to it.

Every now and then, my spam filter and AV fall over, because they run out of RAM. I've attempted to bounce them once a day (via /etc/init.d/foo restart in cron.daily), but that doesn't seem to be working reliably.

When I look at top, mysql is using 118M resident memory. When I bounce it, it goes down to 24M.

So, my question: can I tune mysql to reduce the amount of RAM it wants to use? Or can I get Linux to be more aggressive about kicking it out to swap?

For the geeks out there, I'm on an OpenVZ (Virtuozzo) VPS, and my /proc/user_beancounters shows me going over my kmemsize and privvmpages limits fairly regularly.

I don't have time right now to get the VPS upgraded.

I'd say to try to tweak some of MySQL's tunables. I'm not really familiar with them, but it's probably the best place to modify how it deals with memory. Looks like key_buffer_size and table_cache are appropriate first steps.