That makes me sad. I will miss his voice. I loved how he interrupted his guests and kept them honest and on point. I loved the casual offer for tea/coffee at the end. I would love how it had this encore bit at the end, sometimes!
This podcast chose its listeners and kept it real. Thanks to everyone who makes it possible. Hope they get a fitting replacement for Melvyn and keep it going!
Sometimes I have felt his interruptions to be kind of rude, and have almost sensed an air of irritation by the interrupted guest. But I am not a native English speaker so may have misinterpreted it.
The best interviewers are OK with irritating their guests. An interview guest generally has some conversational line or script that they plan to use. A good host has to break them out of that script - otherwise every interview with that guest will sound the same.
For example, take Demis Hassabis' recent interview with Lex Friedman. Friedman wanted to talk about cellular automata; Hassabis wanted to talk about Gemini AI. Friedman asked questions about cellular automata, Hassabis signalled that he wanted to use his prepared script about AI, Friedman ignored Habbis' signals and continued to ask about cellular automata. The end result is that Lex Friedman listeners get to hear Demis Hassabis talk about cellular automata for 10 minutes - a unique side of Demis Hassabis' mind that you couldn't hear anywhere else. Friedman eventually relented and let Hassabis talk about AI.
(I'm not saying that Friedman is the best interviewer, but he's good. All good interviewers do this: Howard Stern, David Letterman, Charlie Rose, Dick Cavett, etc.)
Basically you have to squeeze your guest to get juice out of them.
We don't really have very many open source models. We have "open weights" models. Ai2 is one of the very few labs that actually make their entire training/inference code AND datasets AND training run details public. So, that this investment is happening is a welcome step.
> So what is final state here for us? Return to menial not-yet-automated work? And when this would be eventually automated, what's left? Plug our brains to personalized autogenerated worlds that are tailored to trigger related neuronal circuitry for producing ever increasing dopamine levels and finally burn our brains out (which is arguably already happening with tiktok-style leasure)? And how you are supposed to pay for that, if all work is automated? How economics of that is supposed to work?
Wow. What a picture! Here's an optimistic take, fwiw: Whenever we have had a paradigm shift in our ability to process information, we have grappled with it by shifting to higher-level tasks.
We tend to "invent" new work as we grapple with the technology. The job of a UX designer did not exist in 1970s (at least not as a separate category employing 1000s of people; now I want to be careful this is HN, so there might be someone on here who was doing that in the 70s!).
And there is capitalism -- if everyone has access to the best-in-class model, then no one has true edge in a competition. That is not a state that capitalism likes. The economics _will_ ultimately kick in. We just need this recent S-curve to settle for a bit.
> Whenever we have had a paradigm shift in our ability to process information, we have grappled with it by shifting to higher-level tasks.
People say this all the time, but I think it's a very short-sighted view. It really begs the question: do you believe that there are tasks that exist which a human can do, but we could not train an AI to also do? The difference between AI and any other technological advancement is that AI is (or promises to be, and I have no reason to believe otherwise) a tool that can be adapted to any task. I don't think analogies to history really apply here.
Oh, I don't know. Maybe build chips that do things 10x more efficiently and sell them a lower cost to compete?
It _is_ a hype bubble but it is also an S-curve. Intel has missed the AI boat so far, if they are trying to catch up, I would encourage them to try. Building marginally better x86 chips might not cut it anymore.
Thats fine. Great even. But thats just normal nueral net inference. Why mention agentic AI over just AI? The gpu doesnt care if the inference is being done for object detection or chain of thought. Intel can only make gpus, their products dont care about the software at the app level.
Maybe they mean the more vram needed for agentic AI? but then the sane thing to say would be that theyll offer more compute for AI.
its just an unhinged thing for a chip manufacturer to say.
But "agentic AI" (and LLMs in general) are far less about compute than everyone talks about IMO. I know what you mean FWIW but she does have a point I think.
1) Context memory requirements scale quadratically with length.
2) "Agentic" AI requires a shittonne of context IME. Like a horrifying amount. Tool definitions alone can add up to thousands upon thousands of tokens, plus schemas and a lot of 'back and forth' context use between tool(s). If you just import a moderately complicated OpenAPI/Swagger schema and use it "as is" you will probably run into the hundreds of thousands of tokens within a few tool calls.
3) Finally, compute actually isn't the bottleneck, its memory bandwidth.
There is a massive opportunity for someone to snipe nvidia for inference at least. Inference is becoming pretty 'standardized' at least with the current state of play. If someone can come along with a cheaper GPU with a lot of VRAM and a lot of memory bandwidth, NVidia's moat is far less software wise than it is for CUDA as a whole. I think AMD are very close to reaching that FWIW.
I suspect training and R&D will remain more in NVidias sphere but if Intel got its act together there is definitely room for competition here.
> Rather than fine-tuning models on a small number of environments, we expect the field will shift toward massive-scale training across thousands of diverse environments.
This is a great hypothesis for you to prove one way or the other.
> Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks.
I am not sure if I buy that, frankly. Even if you were to develop radically efficient means to create "effective and comprehensive" test suites that power replication training, it is not at all a given that it will translate to entirely new tasks. Yes, there is the bitter lesson and all that but we don't know if this is _the_ right hill to climb. Again, at best, this is a hypothesis.
> But achieving this will require training environments at a scale and diversity that dwarf anything currently available.
Yes. You should try it. Let us know if it works. All the best!
This podcast chose its listeners and kept it real. Thanks to everyone who makes it possible. Hope they get a fitting replacement for Melvyn and keep it going!