This workflow is very close to being possible. I gave it a try last year by adding exceptions and test output to clipboard automatically (requires custom code for your stack). The context has increased considerably since my last attempt and agents are now a thing (ReAct loop, etc).
- Integration to your runtime: functions called by the LLM can run your tests, linters, compiler, etc
- Agents: the LLM can define what to do, execute a few tasks, and keep going with more tasks generated by itself
- Codebase/filesystem access: could be RAG or just ability to read files in your project
- Graceful integration of the human in the agent loop: this is just an iteration of the agent but it seems useful for it to ask inputs from the programmer. Maybe even something more sophisticated where the agent waits for the programmer to change stuff in the codebase
My expectation isn’t that the AI generate correct code. The AI will be useful as an ‘agent in the loop’:
- Spec or test suite written as bullets
- Define tests and/or types
- Human intevenes with edits to keep it in the right direction
- LLM generates code, runs complier/tests
- Output is part of new context
- Repeat until programmer is happy