For better or worse it's simply no longer possible to operate a healthcare provider organization using paper records while maintaining compliance with federal interoperability and reporting mandates. That time has passed.
This is currently negative expected value over the lifetime of any hardware you can buy today at a reasonable price, which is basically a monster Mac - or several - until Apple folds and rises the price due to RAM shortages.
$2000 will get you 30~50 tokens/s on perfectly usable quantization levels (Q4-Q5), taken from any one among the top 5 best open weights MoE models. That's not half bad and will only get better!
If you are running lightweight models like deepseek 32B. But anything more and it’ll drop. Also, costs have risen a lot in the last month for RAM and AI adjacent hardware. It’s definitely not 2k for the rig needed for 50 tokens a second
Could you explain how? I can't seem to figure it out.
DeepSeek-V3.2-Exp has 37B active parameters, GLM-4.7 and Kimi K2 have 32B active parameters.
Lets say we are dealing with Q4_K_S quantization for roughly half the size, we still need to move 16 GB 30 times per second, which requires a memory bandwidth of 480 GB/s, or maybe half that if speculative decoding works really well.
Anything GPU-based won't work for that speed, because PCIe 5 provides only 64 GB/s and $2000 can not afford enough VRAM (~256GB) for a full model.
That leaves CPU-based systems with high memory bandwidth. DDR5 would work (somewhere around 300 GB/s with 8x 4800MHz modules), but that would cost about twice as much for just the RAM alone, disregarding the rest of the system.
Can you get enough memory bandwidth out of DDR4 somehow?
Look up AMD's Strix Halo mini-PC such as GMKtec's EVO-X2. I got the one with 128GB of unified RAM (~100GB VRAM) last year for 1900€ excl. VAT; it runs like a beast especially for SOTA/near-SOTA MoE models.
Yes, but that often breaks down once you have bills to pay and need steady income. Real life tends to complicate things. The risk-averse path is a valid choice too, and often the only sensible one.
Plenty of people take risks even with obligations. I am risk averse, and like to ascribe much of that to my obligations to keep my family happy, but really I've always been this way, and would act pretty much the same without them. On the other hand, I've seen folks go to the brink of poverty to start businesses even with small children to feed. They are far more successful today than myself.
There's a video I once saw of Jim Carrey talking about his dad, who, if I recall correctly, was originally a jazz musician, but decided it was too risky and opted to become an accountant instead.
And then, having given up on his dreams to follow the risk-averse path, he got fired.
Jim's words on the subject were like "well, he gave up on his dreams and still failed, so that's why I decided to pursue acting".
reply