Also the area that needs insulating, and in the extremes the amount of air that needs to be exchanged with the outside to make the house livable, and the heat generated by the people living in it (stick a 100W lightbulb in a fridge and see how cold it can get).
The insulation is actually solvable, and for heating can basically remove the power requirements: a house heated and using heat exchange on air leaving vs entering can be heated a lot just by having people inside it, let alone the other energy they use for other purposes. It's just more expensive to build this way, and with cheap energy it can a long time to pay back. Cooling you can't push down past the heat generated inside the house divided by the COP of your cooler, though.
Yep, PassiveHouse standards which typically include an extremely tight envelope which forces installation of outdoor air supply famously can get away with just a few hundred watts of heating capacity because of heat exchange on the incoming and outgoing airstreams!
The insulation is actually solvable, and for heating can basically remove the power requirements: a house heated and using heat exchange on air leaving vs entering can be heated a lot just by having people inside it, let alone the other energy they use for other purposes. It's just more expensive to build this way, and with cheap energy it can a long time to pay back. Cooling you can't push down past the heat generated inside the house divided by the COP of your cooler, though.