The db48x/db50x project seems to be more promising than C47/R47; although, I wish Swissmicros could produce a db50x version with the right layout (no stickies)
Mine is unfortunately now dead - the display has gone dark even though the device is functional. At some point I'll have to look into a repair or replace with something from SwissMicro.
This is me to a big extent. Sometimes I feel like I to learn for learning's sake. Which is okay, or at least that is what my therapist tells me. I struggle with the fact that I "think about doing" vs actually doing.
My work is my hobby too, that is why I struggle sometimes wondering if I will ever retire. Why retire when what I'm doing is for the most part fun. Sure, there are days that I'd rather be "doing X", or more like "studying X" than actually working but I'm enjoying work so much lately that it soon passes.
Work also forces me to actually DO instead of thinking about doing. I have to perform. People are depending on me to get stuff done and that is a big motivator. With my personal projects, no one needs it or is expecting it so it is too easy to abandon.
> I'm not his target demographic
Me either and I am a dev as well
> He's a good presenter and his advice makes a lot of sense.
Agree
Not that I think he forms his answers on who is sponsoring him, but I feel he couldn't do a lot of the stuff he does without sponsors. If the sponsors aren't supplying him with all that hardware then, in my opinion, he is taking a significant risk in buying all of it out of pocket and hoping that the money he makes from YT covers it (which I am sure it does, several times over). But there is no guarantee that the money he makes from YT will cover the costs, is the point I'm making.
But, then again, he does use the hardware in other videos so the it isn't like he is banking on a single video to cover the costs.
gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct.
I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram.
EDIT: Forgot to say what the RAG system was doing which was answering a 50 question multiple choice test about GCP and cloud engineering.
Yup, I agree, easily best local model you can run today on local hardware, especially when reasoning_effort is set to "high", but "medium" does very well too.
I think people missed out on how great it was because a bunch of the runners botched their implementations at launch, and it wasn't until 2-3 weeks after launch that you could properly evaluate it, and once I could run the evaluations myself on my own tasks, it really became evident how much better it is.
If you haven't tried it yet, or you tried it very early after the release, do yourself a favor and try it again with updated runners.
Not parent, but frequent user of GPT-OSS, tried all different ways of running it. Choice goes something like this:
- Need batching + highest total throughoutput? vLLM, complicated to deploy and install though, need special versions for top performance with GPT-OSS
- Easiest to manage + fast enough: llama.cpp, easier to deploy as well (just a binary) and super fast, getting ~260 tok/s on a RTX Pro 6000 for the 20B version
- Easiest for people not used to running shell commands or need a GUI and don't care much for performance: Ollama
Then if you really wanna go fast, try to get TensorRT running on your setup, and I think that's pretty much the fastest GPT-OSS can go currently.
> I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc)
If you share the scripts to gather the GCP documentation this, that'd be great. Because I have had an idea to do something like this, and the part I don't want to deal with is getting the data
Mentions 120b is runnable on 8GB VRAM too: "Note that even with just 8GB of VRAM, we can adjust the CPU layers so that we can run the large 120B model too"
I have in many cases had better results with the 20b model, over the 120b model.
Mostly because it is faster and I can iterate prompts quicker to choerce it to follow instructions.
> had better results with the 20b model, over the 120b model
The difference of quality and accuracy of the responses between the two is vastly different though, if tok/s isn't your biggest priority, especially when using reasoning_effort "high". 20B works great for small-ish text summarization and title generation, but for even moderately difficult programming tasks, 20B fails repeatedly while 120B gets it right on the first try.
But the 120b model has just as bad if not worse formatting issues, compared to the 20b one. For simple refactorings, or chatting about possible solutions i actually feel teh 20b halucinates less than the 120b, even if it is less competent. Migth also be because of 120b not liking being in q8, or not being properly deployed.
> But the 120b model has just as bad if not worse formatting issues, compared to the 20b one
What runtime/tools are you using? Haven't been my experience at all, but I've also mostly used it via llama.cpp and my own "coding agent". It was slightly tricky to get the Harmony parsing in place and working correct, but once that's in place, I haven't seen any formatting issues at all?
The 20B is definitely worse than 120B for me in every case and scenario, but it is a lot faster. Are you running the "native" MXFP4 weights or something else? That would have a drastic impact on the quality of responses you get.
Edit:
> Migth also be because of 120b not liking being in q8
Yeah, that's definitely the issue, I wouldn't use either without letting them be MXFP4.
Hmmm...now that you say that, it might have been the 20b model.
And like a dumbass I accidentally deleted the directory and didn't have a back up or under version control.
Either way, I do know for a fact that the gpt-oss-XXb model beat chatgpt by 1 answer and it was 46/50 at 6 minutes and 47/50 at 1+ hour. I remember because I was blown away that I could get that type of result running locally and I had texted a friend about it.
I was really impressed but disappointed at the huge disparity between time the two.
I used pg vector chunking on paragraphs. For the answers I saved in a flat text file and then parsed to what I needed.
For parsing and vectorizing of the GCP docs I used a Python script. For reading each quiz question, getting a text embedding and submitting to an LLM, I used Spring AI.
It was all roll your own.
But like I stated in my original post I deleted it without backup or vcs. It was the wrong directory that I deleted. Rookie mistake for which I know better.
Then trying to get drunk or laid took over and I only dabbled with it here and there. Got married, had kids, did other things and might as well say abandoned it.
For about the past 10 years I've been doing programming nearly every day.
I wish from circa 1993 to 2010 I had been more heavily involved with it than I was.
I finally have four ideas that I think worthy to build that I would like to monetize. All would be well within my abilities to build. No vision of grandeur that I'd retire from any of them and if I made $100 from one site I'd be ecstatic.
Two are simple games, one a directory and one a utility type site. No AI, no sign-up, no affiliate marketing, no upselling, just simple sites with ads.
However, my "paralysis by analysis" affliction is strong.
> However, my "paralysis by analysis" affliction is strong.
The solution is to remember that nothing is perfect, and that all code is eventually thrown away and replaced. So just start writing code and have fun!
Strongly agree. I struggled with the same. A key discovery for me was that this "paralysis by analysis" is a form of perfectionism.
I always thought perfectionism was someone who was productive but way too hard on themselves, overworking to achieve some end that's just not worth it. Not necessarily - it can also mean DOING NOTHING because you dont see a way to do it perfectly.
This has helped me a lot with writing. Sometimes you just have to write down incoherent slop. Let the ideas flow and be content with knowing they will have to be revised later. By all means if you write with more purpose and structure without getting too bogged down to continue then do so.
I think you have the motivation wrong. The cost-benefit economics work if you want to have fun building something, learn a lot and share it with others. It doesn't work if the goal is to make up to $100/month selling ads; getting a part-time job would be a better path. In this scenario not finishing your side project is the correct decision, and not starting an optimization of that.
To me, trying to make money with random projects is the most motivating thing. A dollar earned from some little project is emotionally to me worth many times that of the same dollar I'd earn as salary (as long as I don't starve). Most of my friends do not seem to share this feeling.
Also the internet is very big. You can sometimes have success with something, even it's a very silly badly implemented little thing. What people like, how you happen to get traffic, it's all quite unpredictable.
I'm right there with you, except at times I have thrown caution to the wind and made my sites available.
My current setup is to rent a cheap $5/month VPS running nginx. I then reverse ssh from my home to the vps, with each app on a different port. It works great until my electric goes out and comes back on the apps become unavailable. I haven't gotten the restart script to work 100% of the time.
But, I'd love to hear thoughts on security of reverse SSH from those that know.
I do something similar with my home server, but with a WireGuard split tunnel. Much easier to set up and keep active all the time (i.e., on my phone).
Nginx handles proxying and TLSing all HTTP traffic. It also enforces access rules: my services can only be reached from my home subnet or VPN subnet. Everywhere else gets a 403.
Why not just have nginx listen on the Wireguard interface itself? That way you drop all traffic coming inbound from sources not on your Wireguard network and you don't even have to send packets in response nor let external actors know you have a listener on that port.
Maybe try running your services in docker, I don't know how difficult that would be to implement for you, but if you run it in containers you can get it to start up after an outage pretty reliably.
If you need a middle ground between docker and k8s, you might have a look at nomad. Definitely a learning curve, and I find the docs lacking, but easier to set up and maintain than k8s.
Correct, there is no public IP address exposed to my home.
Right now my "servers" are Dell micro i5s. I've have used RPI 3 and 4 in the past. My initial foray into self-hosting were actual servers. Too hot, too noisy and too expensive to run continuously for my needs, but I did learn a lot. I still do even with the micros and pis.
What do you use for your remote server? Because even a VPS seems kinda overkill, if all it's doing is some redirecting. I guess you could do TLS termination there aswell...
Every time I use for more than a couple of calculations I think how much I prefer a RPN calculator.