Sounds to me like your problem isn't your team size. It's your team members (or some of them). Agile only works if there's unit tests everywhere (otherwise you can't refactor, and if you can't refactor then you can't evolve the good design that you didn't do up-front). Software development in general (even before Agile) only works if code is modular. If one or more of your team isn't doing those things, you're sunk unless you can fix it...

If you hired a bricklayer and got one who didn't put mortar between his bricks because it's quicker, and whenever QA leant on one of his walls it fell over, and you told him about the mortar thing and he still didn't bother, then I think you'd very soon get a new bricklayer. Software without modularity, and particularly faux-Agile software without unit tests, is at least as likely to fall over as a brick wall without mortar.

As this is the Empeg BBS, I should say that the car-player itself was not developed using Agile, and at that time there were basically no unit tests in the code. We got away with that, I think, because the original team was unusually tight-knit and unusually on-the-same-page about design and vision; as the team grew in the Rio/Sigmatel era and became less homogeneous, unit testing became essential. (It was JG who was our first great evangelist of unit tests, not me, but it was soon clear to everyone what a benefit they were.)

Peter