how much bass should I reasonably expect to gain from installing an in-ceiling speaker vs open-air.
Something I recall from my past days of
IASCA car stereo competition might (or maybe not) be relevant here. (IASCA: International Auto Sound Challenge Association)
The lowest frequency you will attain is a function of the amount of cone area in a given enclosure. For example, a 12 inch subwoofer has a cone area of ~113 square inches. Compare to a pair of 10" speakers in a common enclosure at ~157 square inches, an increase of nearly 40%. You can attain lower frequencies with a pair of 10's than with a single 12. But that's not the biggest advantage. With the 12's you are pretty much constrained to a 20 Hz crossover frequency. The 10's are perfectly happy with a 100 Hz crossover, giving you a "richer" bass sound. Not just "blumph-blumph-blumph" like you her from the ricers' stereos, but actual music from the subwoofer box, with very tight "pomp-pomp-pomp" bass hits.
I won every IASCA competition I entered with that setup.
tanstaafl.