as soon as I would do that the voltage would drop to almost nothing.
... Ive got it working, but with no safeguards.
Maybe I bought the wrong kind of capacitors?

hmmm, caps on the input to ground and output to ground should not affect voltage, just stability, unless they're shorted internally... just about independent on type.

/Michael