This is something that you can bootstrap into a proof-of-concept in a day and learn the tools you like and don't like along the way.
Basically you'll use any LLM and a vector DB of your choice (I like ChromaDB to date). Write a tool that will walk your source documents and chunk them. Submit the chunks to your LLM with a prompt that asks the LLM to come up with search retrieval questions for the chunk. Store the document and the questions in ChromaDB, cross-referencing the question to the document source (you can add the filename/path as metadata to the question) and the relevant chunk (by it's ID).
Run this tool whenever your docs change - you can automate this. Being intelligent about detecting new/changed content and how you chunk/generate questions can save you time and money and be a place to optimize.
To use it, you need to accept user input, run the input as a text query against your vector DB and submit both the results (with filenames and relevant chunks) and the user's query to a LLM with a prompt designed to elicit a certain kind of response based on input and the relevant chunks. Show the response to the user. Loop if you want.
You can build most of this with as few tools as `litellm`, `langchain` and `huggingface` libraries. You'll be surprised how far you can get with such a dumb setup.
Yes, this is basic RAG. That's how you do it without getting overwhelmed with all the tooling/libraries out there.
> This is something that you can bootstrap into a proof-of-concept in a day and learn the tools you like and don't like along the way. Basically …
Where do sloppyjoes get all of this unrestrained optimism?
OP asked if such a solution exists.
This documentation assistant is an oft requested tool in regards llms on this forum. If you can do it, you could start a business around it. OP could be your first customer!
The only other comment in this thread at this time is from someone who is also a breathlessly vocal supporter of contemporary machine learning systems on this forum and yet they are saying “I have yet to see a convincing demo”, but here you are saying it’s easy; if only these damned margins were larger!
I’ve checked your GitHub. I’m unable to find an implementation of this thing that you claim is so simple to implement.
I checked your blog. Your most recent article is about you wasting 45 minutes hoping such an “ai agent” can fix a bug in your code. It proved unable to do so. You even call the experiment a failure in your post.
So, where’s this optimism coming from?!
But you do say you are having fun. Which is great! I’m glad you’re having fun.
You misinterpreted my comment there. When I said "I have yet to see a convincing demo" I was talking about the idea of fine-tuning a model to answer questions against documentation. The rest of my comment described RAG - the exact same approach that tgittos is recommending.
Here are a few of my own RAG implementations - getting a basic version working really is something that can be done in a few hours... but getting a GOOD version working takes a LOT longer than that.
It isn't perfect and audio occasionally de-syncs or a speaker drops out. On a dedicated wireless channel playing a local source I imagine it would be pretty solid.
Not really an option if you don't have a Mac to run Airfoil, though if you do you can use the Satellite (https://rogueamoeba.com/airfoil/satellite/) program to sync Mac your other non-Apple devices to the host.
I moved out of Australia and let my bank know that I would likely be using credit cards overseas for a while.
Sure enough, as I was trying to buy furniture for my new place, the same thing happened to me. They blocked the card and when I called them they scolded me for not letting them know that I was traveling. I told the agent that I did in fact let them know, and asked them to update the account to note that I lived overseas.
Some 5 years later I needed to use one of those accounts again. I diligently phoned the bank, let them know I lived overseas and that I was going to use the card. I used the card, the transaction was blocked, I called and was again scolded for not telling them that I went on holiday.
I swear the 'notes' they add to your account are just an eye-roll and then a call disconnect.
I started building a Magic based data science tool for 6 months before coming to the same inevitable conclusion. It sits in a Github private repo rotting away.
There is nothing wrong in being prepared for things to go wrong.
I’m always prepared for my car breaking down on every trip I make, no matter how short. It’s not that I don’t have faith in my car and driving, it’s that I don’t trust every other variable in the equation.
Respectfully, this is not being prepared. This is being too paranoid which is exactly what you cannot do if you want to start your own thing and take a risk. There never is a perfect time to do your own thing.
Knowing your options and determining potential risk to your backup plan isn’t paranoia.
My car already has the few things I need in the trunk. All I need to do is check the weather and choose what clothes to wear. Having a road atlas for the entire United States means I can navigate without a phone. (Which has happened.) Having an extra hoodie and a pair of old boots means I can walk several miles in snow.
All of these items have been used or badly needed the past.
If I was starting a company, I’d do similar things and I wouldn’t worry about failure and would take risks, even serious ones, with confidence.
Mentally preparing for failure is a perfectly reasonable thing to do, but it's not about "feeling sad" and "giving up", but about thinking through all the things that may ruin your business. You'll also be less shocked when you do fail.
To be a bit of a downer, but also not sugar-coat it, I'd assume there's a higher chance of death from taking a big risk and losing than there is from driving your car on regular roads also.
I don't know of any examples of founders specifically, but it's very common with gamblers/traders.
I'm not sure I'm following this line of thinking. Are you saying that the best way to handle risks is to put your head in the sand and pretend they don't exist?
If I was thinking about driving through the Amazon I would ask others what the experience was like to better understand the risks, not just hop in the car and hope for the best.
> If I was thinking about driving through the Amazon I would ask others what the experience was like to better understand the risks, not just hop in the car and hope for the best.
Not what you are doing.
If you were probing to discover risks, you wouldn't be here asking what happens if your business fails. You would be asking how to prevent the business from failing.
> Are you saying that the best way to handle risks is to put your head in the sand and pretend they don't exist?
When it comes to entrepreneurship, yes.
You don't know what will come along and kill your business. Maybe Meta eats your lunch. Maybe the government in your jurisdiction passes a law that is incompatible with your primary monetization strategy. Maybe your house burns down and you need to sell.
Are any of these actually worth worrying over and planning mitigation against? Or are they your own personal black swan events that you just need to handle in the moment?
You know when the best time to worry about getting a job after your startup fails? When your startup fails.
To put it bluntly, this is all about mindset. You can't de-risk starting a company. You can only believe in your ability to overcome the obstacles that will come up as they come up. Planning what happens if you fail shows enough lack of belief in yourself that you almost de-facto will fail, because that exit door is always there if things get hard.
I didn't mention it in the post, but this was more so coming from an angle of preparing for money running out. How much runway is enough knowing that if I fail there will be some amount of lead time to find a new gig and if that new gig will pay similarly to what I am making now.
That will depend on things you won't know until you start.
How much do you start with, what's your burn rate, how long until you get revenue, are you bootstrapping/funding, your personal cost of living, etc.
Running a business involves risk. Risk to your comfort, your mental health, physical health, financial health, etc. Closing a business is the end of all of that, not the start.
Why do you want to scratch that entrepreneurial itch?
I've tried everything you have (except weather stations and mechanical pencils) but also make/play guitars, hobbyist electronics, paper making, painting/illustration, fixing up old cars, and MtG among others.
I realized I was being a little ridiculous when I started planning a peltier powered hydroponic cabinet for growing fresh wasabi. Most of that was to do with the disappointing results with peltier cooling temperature deltas.
The garage full of specialist tools is what kills me. I can't imagine how I'll ever use the vacuum chamber I built for epoxy stabilization again, but I have one.
I reused a fermentation setup for the hydroponic wasabi idea. I had some success with lettuce as a test subject but the humidity was not well-controlled which I think would be a problem. The thought of how much energy I'd have to use for years to keep the whole thing running just made it seem a bit ridiculous. Also, I'm not sure if I could reliably source wasabi seeds. I stick with super-hot peppers mostly which don't need a lot in the way of temperature control.
I reached a similar conclusion, luckily before doing too much building. I'm happy with terrariums so far, they scratch my 'living things in closed systems' itch.
Funny, I was sort of exploring the idea of aquaponics myself and I was trying to explain to someone what was at the root of my curiosity and I realized that it was enclosed ecosystems. I had fun giving the lettuce to people and explaining that it was grown without ever seeing natural sun or soil and showing them a timelapse video of the growth, but no one seemed all that impressed lol.
If you've never been, I can highly recommend a trip to "Biosphere 2" in Arizona which is a semi-retired monument to the whole idea, and quite worth the tour.
Keep leaning into it. Each new capability is a gateway to more. Now that you have rough vacuum you can start doing plasma experiments. That comes wrapped up with a bunch of lab / high voltage safety stuff, but also enables x-ray experiments. Could then branch off into intersections with crystals/minerals/chemistry. Anyway that is just one branch.
This is exactly the boat I'm in. I have a 1hr train commute to work that I spend skilling up in AI. I've been following the space for about 15 years and have done a bunch of self learning of earlier ML techniques (the early Stanford ML MooCs) so I'm not coming in cold. What I'm doing is:
I would agree with other commenters that recommend learning how to implement a paper. As someone who barely managed to get their undergraduate degree, papers are intimidating. I don't know half the terms and the equations, while short, look complex. Often it will take me several reads to understand the gist, and I've yet to successfully implement a paper by myself without checking other sources. But I also know that this is where the tech is ultimately coming from and that any hope of staying current outside of academia is dependent on how well I can follow papers.
I've been doing this for about a month now, and I feel I definitely understand more of the theory of how most of this stuff works and can train a simple attention based model on a small-ish amount of data. I don't feel I could charge someone money for my skills yet, but I do feel that I will feel ready with about 6 months - 1 year of doing this.
Basically you'll use any LLM and a vector DB of your choice (I like ChromaDB to date). Write a tool that will walk your source documents and chunk them. Submit the chunks to your LLM with a prompt that asks the LLM to come up with search retrieval questions for the chunk. Store the document and the questions in ChromaDB, cross-referencing the question to the document source (you can add the filename/path as metadata to the question) and the relevant chunk (by it's ID).
Run this tool whenever your docs change - you can automate this. Being intelligent about detecting new/changed content and how you chunk/generate questions can save you time and money and be a place to optimize.
To use it, you need to accept user input, run the input as a text query against your vector DB and submit both the results (with filenames and relevant chunks) and the user's query to a LLM with a prompt designed to elicit a certain kind of response based on input and the relevant chunks. Show the response to the user. Loop if you want.
You can build most of this with as few tools as `litellm`, `langchain` and `huggingface` libraries. You'll be surprised how far you can get with such a dumb setup.
Yes, this is basic RAG. That's how you do it without getting overwhelmed with all the tooling/libraries out there.