There are several things wrong with your assessment. First, far more heat is lost through conduction than radiation: the outside air touching your house, especially your windows. Second, you fail to take into account the energy required to reheat the walls, which have far more mass than the air in the house. If you're reheating the walls every afternoon when you get back from work, rather than minimizing the temperature difference between the walls and the (interior) air.

I'll admit that these calculations may be very different in significantly colder climates.