Cool to see that this worked well for someone. Super hard to force the key insight in a problem to magically appear given more time sunk into it. Big weakness of mine honestly, and requires a lot of self-awareness to pull myself out of a problem-solving rut. I like the idea of hacking sleep - do you find yourself priming your mind with the problem before nodding off? Curious how a bedtime wind-down routine factors into how effective this is.
Over years of math undergrad and grad school I tried very hard and was never able to get this to work, so you're not alone. I was able to reliably reproduce hopeful feelings after sleep, but upon investigation the "new leads" were either things I had already tried (and forgotten why they didn't work) or they were the type of imprecise high-level vague direction ideas that were never difficult to generate and still had 99% of the true effort remaining to grind through the details.
Very cool. I used to dream of this stuff when I was younger. Reminds me of Atlantik Solar: https://www.atlantiksolar.ethz.ch/. Hasn't been updated in a while, but focused more on low-altitude autonomous survey missions.
> I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback.
I guess if we ignore pretraining, don't sample-efficient fine-tuning on carefully curated instruction datasets sort of achieve this? LIMA and OpenOrca show some really promising results to date.
distilbert was trained from Bert. there might be an angle using another model to train the model especially if your trying to get something to run locally.
Yep, batching is a feature I really wish the OpenAI API had. That and the ability to intelligently cache frequently used prompts. Much easier to achieve this with a hosted OS model, so I guess it's a speed + customizability/cost tradeoff for the time being.
imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from multiple customers in it
