TL;DR: AutoGPT has similar if not identical problems
re: AutoGPT
Got really excited at first. Thought maybe I had missed that it was viable in a year of playing with LLMs. Tried a few demos, didn't work. Looked into it more and confirmed a core loop involved LLM eating its own output to make decision.
I still kept investigating it on my todo list, in case my earlier experiments with that approach were wrong.
I took it off the todo list later.
I saw near-universal feedback like yours, that general technique never worked IMHO, and IMHO sycophancy explains why. Paper here[1], TL;DR the model is very likely to agree, so critical feedback loops over multiple steps tend to settle into a loops of steps.
IMHO this doesn't mean sycophancy breaks _all_ workflows, ex. a flow for writing a story involving outlining, writing, criticizing, then rewriting is a genuine real quality boost.
However, if a human does write => criticize 10 times, it keeps getting better each iteration. If you have an LLM do it 10 times, IMHO it's actively harmful after round 3.
"outline" => ["write page 1", "write page 2", "write page 3"] => ["feedback on page 1"..."feedback on page3" => ["use feedback and original draft to rewrite page1"..."page3"] => "combine pages 1 2 and 3 into cohesive story"
I've tested the writing loop you describe and the biggest problem seems to be that currently GPT4 is good at generalities, but gets into loops of generalities that doesn't drive things forward when you expect too much detail.
It's not that it can't improve the writing further per se, but that it takes very detailed prompting to get it to give a detailed enough critique to do so consistently enough across even a page (e.g. ot might come uo qith a great lone but proceed to edit out the best paragraph elsewhere) to the point that I tend to agree with you in as much as it at least will take a much more convoluted chain of prompts to maybe get there at the moment, and you'll be fighting GPT4s tendency to actively cheer on really juvenile prose the whole way.
In a way I think the biggest hindrance to get it to write better at the moment is that it has awful "taste", and having to explicitly give it a long list of rules to check against is a poor substitute.
As an aide, though, I think you could get reasonable but not great results at "bridge these two paragraphs and maintain the style" type tasks, or expanding descriptions into a paragraph or two, though more so for non-fiction writing.
For fleshing out the basics of a technical spec and pointing out what I've missed I've had decent luck, on the other hand. It's not come up with any Earth shattering revelations, but for a dry spec that's not the point.
> However, if a human does write => criticize 10 times, it keeps getting better each iteration. If you have an LLM do it 10 times, IMHO it's actively harmful after round 3.
>> The game has no winner: the entertainment comes from comparing the original and final messages. Intermediate messages may also be compared; some messages will become unrecognizable after only a few steps.
>> The transmission chain method is a method used in cultural evolution research to uncover biases in cultural transmission.[1] This method was first developed by Frederic Bartlett in 1932.[2][1]
> In mathematics, computer science and logic, convergence is the idea that different sequences of transformations come to a conclusion in a finite amount of time (the transformations are terminating), and that the conclusion reached is independent of the path taken to get to it (they are confluent).
> More formally, a preordered set of term rewriting transformations are said to be convergent if they are confluent and terminating.[1]
To simulate a full-scale multi-agent game, there would need to be a "Fourth Estate" (and maybe a Fifth Estate); a peanut gallery of dissenters with signs and no jobs.
Then, you could model consensus with LLMs and have something better than the mediocrity that sometimes results from committees.
More samples from the same LLM vs Sample different LLMs
Maybe feed one or more agents relevant encyclopedia articles as context first; which read the most encyclopedia articles first?
We tried it recently and found it very challenging to get it to accomplish simple tasks.