You are naive if you think you have 10-15 years. GPT-5 will most likely be out by the end of the year. It will be significantly better than GPT-4. I expect it will replace millions of people - not explicitly - companies will gradually have fewer people doing more work, resulting in increasing layoffs and decreasing hiring. This has already started happening: I spoke to several startup founders recently who use GPT-4 instead of hiring marketing people.
GPT-5 will be significantly better at coding, to the point where it might no longer make any sense to hire junior developers.
And this is just GPT-5, this year. Next year there will be GPT-6, or an equivalent from Google or Anthropic, and at that point I fully expect a lot of people everywhere getting the boot. Sometime next year I expect these powerful models will start effectively controlling robots, and that will start the process of automation of a lot of physical work.
So, to summarize, you have at best 2 years left as a software engineer. After that we can hope there will be some new types of professions that we could pivot to, but I’m struggling to think what could people possibly do better than GPT-6, so I’m not optimistic. I’d love for someone to provide a convincing argument why there would be any delay to the timeline I outlined above.
p.s. I just looked at the other 20 responses in this thread, and it seems that every single one is based on current (GPT-4) LLM capabilities. Do people seriously not see any progress happening in the nearest future? Why? I’m utterly baffled by this.
> And this is just GPT-5, this year. Next year there will be GPT-6, or an equivalent from Google or Anthropic, and at that point I fully expect a lot of people everywhere getting the boot. Sometime next year I expect these powerful models will start effectively controlling robots, and that will start the process of automation of a lot of physical work.
> So, to summarize, you have at best 2 years left as a software engineer. After that we can hope there will be some new types of professions that we could pivot to, but I’m struggling to think what could people possibly do better than GPT-6, so I’m not optimistic. I’d love for someone to provide a convincing argument why there would be any delay to the timeline I outlined above.
This reads to me exactly like people who said learning to be a truck driver in the early 2010s was stupid because we were 2-3 years away from self driving trucks taking their jobs. I have no doubt that the models will get better, but being 90-95% right still implies you need people for the last 5%. I think, like self driving, the corner case 5-10% is going to be really really hard to iron out and it will not be ironed out in 1-2 years like your comment says. We only just barely have self driving taxis now (despite them being 1-2 years away for the past decade and a half), and we have no self driving long haul trucks afaik.
1. Deep and detailed understanding of how the world works. We are just starting to make real progress there (GPT-4), and more work is needed [1].
2. Reliability. A model should make significantly fewer mistakes than humans would make in similar scenarios, on average. This includes factual and logical mistakes, as well as hallucinations.
I expect the main improvements GPT-5 will bring are improvements in exactly these two areas. The first one is likely to come from training on huge video datasets (next frame prediction objective), and the second one will require high quality data, and some other methods (known and secret), but given that OpenAI has stated many times in the last year that improving reliability is their number one priority, I believe we will see a significant improvement there. Note that simply being better driver than humans is a very low bar, and to be accepted/adopted the self-driving AI must be much better (10x or even 100x better). But I believe that even today’s technology (such as the best models from Waymo or Tesla) could be used today in long haul trucking with similar or better accident rates. And this technology is not even based on large foundational models like GPT-4. Obviously the necessary regulation will delay the automation of self-driving trucks, that’s why I said the automation of physical jobs will come after the automation of routine office jobs like (most types of) software engineering.
Other than those two challenges, there’s also an engineering challenge to put a GPT-5 scale model inside every car (needs to run locally). This can be achieved by producing custom built hardware accelerators, but will still be expensive in the near term, so I expect that self driving will become widespread after the cost of a computer inside every car falls below 10% of the cost of the car. Currently I’d imagine we would need an equivalent of an 8x H100 server to run a highly compressed and finetuned for driving version of GPT-5.
That may all be true, but it sort of sidesteps my point. My point is that people have been saying for 15 years that "technology X" will cause truckers to become obsolete and create ubiquities self driving cars, and there has been billions (if not trillions) of dollars poured into it, and it has not come to fruition (yet). I do think it will eventually be automated, but your comment says we have "at best 2 years left as a software engineer." and that seems very naive to me given that we have seen your exact same argument for the past 15 years. Lets even imagine the tech gets to the point where software engineers can be fully automated, as you mentioned above, regulatory hurdles will need to be crossed even for office jobs and I just don't see that happening in 2 years. I do think it will happen, but if self driving is any indicator, it will take a couple decades at least for the tech and regulatory hurdles to be overcome.
The difference between our opinions is I see 2024 as the time just before the knee of the exponential progress curve, and you see it as a long way to go to get to the knee. I realize how saying "this time it's different" might sound. But I do think this time it's different.
I remember when I read http://karpathy.github.io/2015/05/21/rnn-effectiveness back in 2015 I became convinced that these models are scalable. I remember thinking if only we could find a way to train a really big RNN/CNN hybrid on a lot of video data to try to predict the next video frame we would eventually force it to develop understanding the world. Predicting what happens in a video frame is a lot harder than predicting the next word (just ask Lecun), but it turned out that even just predicting the next word is extremely effective, and GPT4 feels like the first model that finally "understands" the world. To me, this was the hard part, developing this proof of concept that we can get there simply by scaling. Next step is video prediction, and we have a lot of room for further scaling to get there. There is a lot of video training data, and we can scale our models a lot more. The progress is mainly limited by available hardware processing power. There's no lack of good ideas to try to make things work.
In a way, 2024 feels like 2012, when deep learning took over ML world by storm. The same thing is happening now with multi-modal foundational models. GPT4 is like AlexNet - a culmination of many years of gradual improvements, a combination of unprecedented scale and various tricks. Think about every improvement starting with GPT1, which established state of the art in language modeling using a simple, universal, and scalable model architecture. GPT2 was able to generate a high quality one page long text. It's funny, it does not even sound that impressive now, but at the time it was absolutely mind blowing. GPT3 demonstrated incredible generalization capabilities, and significantly raised the quality and reliability of generated output. GPT4 took it to another level, achieving human-level reasoning capabilities. Every single one of these breakthroughs took me by surprise, and I do deep learning research for a living. I have absolutely no reasons to believe we have reached a saturation point in quality of these models. So what's next? Where do we go from already near human capabilities of GPT4?
What do you expect from GPT-5? In what ways do you think it will be better than GPT4, and what will be its main limitations? Which aspects of software engineering do you think it will excel at, and which aspects we will still need humans for? Would these challenging aspects still not be solved in GPT6, assuming another significant improvement in quality over GPT5? I will not be surprised if GPT6 will be designed by GPT5, with some help from humans. How does your timeline of AI progress look like?
Yes, me too! I am surprised people are unaware of the empirical power-scale law. [1] Currently, Anthropic's new model has proven it by being better than GPT-4 on some tasks. Yet, the old people are saying it is okay. They are saying engineering is not just coding. What they don't seem to grasp is there won't be a necessity for as many engineers.
If people are not willing to accept juniors, where will the next generation of seniors come from? No clue about that. Probably, that spot is reserved for the brilliant minds graduating from R1 institutes. What about the average Joe? Not everyone has the fortune to work on very specialized domain skills. People just don't get it easily.
Everyone in academia has admitted to productivity boost by 30-40%. And here we are talking about hard research. Imagine, what it looks for regular job which is managing and maintaining codebases. The severity of this situation is truly alarming not for the greybeards but for us, Zoomers. With climate crisis, AI, geopolitical instability, it feels really great to be alive.
GPT-5 will be significantly better at coding, to the point where it might no longer make any sense to hire junior developers.
And this is just GPT-5, this year. Next year there will be GPT-6, or an equivalent from Google or Anthropic, and at that point I fully expect a lot of people everywhere getting the boot. Sometime next year I expect these powerful models will start effectively controlling robots, and that will start the process of automation of a lot of physical work.
So, to summarize, you have at best 2 years left as a software engineer. After that we can hope there will be some new types of professions that we could pivot to, but I’m struggling to think what could people possibly do better than GPT-6, so I’m not optimistic. I’d love for someone to provide a convincing argument why there would be any delay to the timeline I outlined above.
p.s. I just looked at the other 20 responses in this thread, and it seems that every single one is based on current (GPT-4) LLM capabilities. Do people seriously not see any progress happening in the nearest future? Why? I’m utterly baffled by this.