Speculation affects housing prices only because of development restrictions. Because only a small fraction of land can legally be developed, it's possible to corner the market for new development. If all property can be developed, there would be orders of magnitude more competitors in the development space, and impossible for an individual or cartel to control.