My Question is this: is soldering a big deal? I never soldered any connections on my unit and I'm pleased with the sound quality. Would it make a difference at all?

Soldering isn't so much about sound quality as it is about durability and longevity. If you just twist wires together, they're going to fall apart pretty quickly. If you use a crimping tool and some crimping connectors, that will buy you some time, but those will get corroded over time, and they can come apart if the wires get tugged.

The quality thing comes into play when you try and pour a bunch of juice through the wires. If they're just crimped or twisted, high-power applications will suffer because those kinds of connections won't transmit enough electrons to get the job done properly.

[anecdote]
I have a friend who's quite experienced in sound/audio stuff. He's a recording engineer and a music producer, as well as a musician. However, he didn't have much car audio experience when he installed a certain car stereo a few years back. Since my family often went on trips with his family, I spent a lot of time in the back seat of this particular car. And I kept telling him that his right rear speaker would cut out every time he went around a corner a certain way. He didn't believe me because he didn't notice it from the front seat for the brief periods it happened. And he was so sure that the crimp connectors on those speaker wires were fine. Until they just fell out completely and the speaker quit working altogether...
[/anecdote]

[soapbox]
I am a firm believer that all car stereo wiring should be carefully soldered and shrink-tubed (note: not just wrapped in tape because that comes undone, too). If you're not prepared to solder and shrink-tube the wiring connections on your car stereo installation, it's time to hire a professional to do the job. Period.
[/soapbox]

___________
Tony Fabris