We're running out of the ability to make transistors smaller and closer together so beyond some major breakthrough I wouldnt expect Moore's law to continue nearly long enough to get to the point of running GPT4 on consumer hardware in the short term
Well consumer hardware can run something in the order of ~50B quantized at a "reasonable" price today, we'd need about 5 or 6 doublings to run something that would be GPT 4 tier at 1T+. So, it would need to continue for roughly a decade at least?
Current models are horrendously inefficient though, so with architectural improvements we'll have something of that capability far sooner on weaker hardware.
That doesn't solve the problem, it just pushes is down the road a bit. The exponential growth is merely offset by a constant factor once. Unless we figure out how to push transistors in the 5th, 6th etc dimension with every new generation.
It was never a solution, Moore's law has more than one dimension as well, not just density but heat dissipation. Can't cool down a transistor that's surrounded by transistors on all sides.
