Sounds to me like your problem isn't your team size. It's your team members (or some of them). Agile only works if there's unit tests everywhere (otherwise you can't refactor, and if you can't refactor then you can't evolve the good design that you didn't do up-front). Software development in general (even before Agile) only works if code is modular. If one or more of your team isn't doing those things, you're sunk unless you can fix it...
If you hired a bricklayer and got one who didn't put mortar between his bricks because it's quicker, and whenever QA leant on one of his walls it fell over, and you told him about the mortar thing and he still didn't bother, then I think you'd very soon get a new bricklayer. Software without modularity, and particularly faux-Agile software without unit tests, is at least as likely to fall over as a brick wall without mortar.
Peter
This is a fair assessment- but I content that size has something to do with getting away with it. That is, one of the things about Agile is that visibility is so high that you either get with the program or you ship out. However, there are just a ton of moving parts with the way things are currently structured and differences in development get chalked up to "difference in style" rather than "weakness". I actually told the project manager several months ago that I felt that developer in question had a "lack of skill" and that effectively ended our conversation on the subject.
I'll also admit, if I can get the people not writing unit test working on an area of code that I don't have to work on, at least the effects are isolated and when we start having quality issues we can consider a re-write for just that area. As it is we just have a general weakening of our code base and it tends to affect all of our development.
One problem is that the developers from my company are huge believers in unit tests and TDD, whereas the contractors are not. We have bent over backwards not to draw lines and make the dynamic "us vs them"- but it's really hard when we are consistently writing unit tests and they are not. Anyway, I've been beating this drum for months now and I think I pretty much annoy people at this point when I bring up unit tests- but what am I going to do?
My boss's answer to the issues we are having is to put measures in to enforce good unit testing (the lone exception to the "I don't write unit tests" rule on the consulting side is the TA who works for them- so he'll agree with the measures). I hope that helps. I just feel that if you've told people what to do for a year and they continue to not do it, FORCING them to do it isn't going to really fix things. I want team players who want to work together, not people who want to do it their way and ignore the team. I ALSO think that in smaller teams it would be easier to address this stuff at the individual level rather than making sweeping rules that can easily lead to a hostile environment.