So I take my roughly $5000 and start an investment account that will return 4.625% annual interest compounded monthly.

If that's all you're getting on your investment account, you need to consider some different investments. The S&P500 averages out to about an 11% gain per year, despite the bubbles and busts. While I think it's entirely possible to beat that (my annualized 401k returns are several percentage points higher than the S&P500 returns), you certainly shouldn't be doing any worse than the market. Try running those numbers again with 11%, and see how much cash you end up with.

In my situation, I'd be an idiot not to buy.

Agreed. Wish I could do the same here in LA.