When you take a picture with a camera, the picture is distorted. Usually it's barrel distortion, so called because lines that were straight in real life bow outwards (that is, towards the edge of the photograph) at the middle, or pincushion distortion, which is the same thing but with the curve in the opposite direction. (Barrel distortion should really be called pillow distortion or something, as it happens in both/all axes, and barrels don't have rounded tops and bottoms. But that's neither here nor there, not to mention that examining the distortion in only two axes is a gross oversimplification to begin with.)

Since straight lines are no longer straight, but curved, and you're taking multiple pictures with different origins of the distortion, it's unlikely that things will line up properly, as the offset of an object in two different photographs is unlikely to be the same. Theoretically, I suppose, if you could precisely measure the rotation of your camera and make all of the photographs radially equidistant, you could line the photographs up so that objects would appear to be nondisjoint by making sure that the join happened at the exact middle point between the distortion origins. But that sounds impractical at best. Even then, though, you'd still have curvature of straight lines, so what was a straight horizontal line all the way around you in real life would appear scalloped in the photograph.

Commercial software deals with this by first dealing with the optical distortion so that things that were straight in real life are straight in the photograph. It's not perfect, but it's close.

There's more information on the Wikipedia page about Optical Distortion.