I know. But here where I am, using torrent means participate in distribution of the content and that is where I'll get huge bill for illegally sharing this file.
There is one trained on 600B tokens from SlimPajama [1], but that's fairly tiny compared to other recent releases (ex. stablelm-3b [2] trained on 4T tokens).
> low quality data (the Pile only)
The Pile is pretty good quality wise. It's mostly the size (300B tokens) that's limiting.
Eh quality is subjective. There are good parts, like Books3 and arxiv, but a large part of it is common crawl which has just about anything people put up on the internet, random IRC chat logs, HN and Reddit shitposts, Youtube subtitles which are in broken English half the time, and of course the Enron corporate email dump to make every model sound like an HR middle manager.
It has become harder and harder to do so, because Twitter (in its default state) now insists on inserting brain-meltingly lowbrow content in between every other post from someone you actually follow. And adverts for low quality iPhone games that are themselves also simply platforms for further adverts.
This is no longer true. Twitter now has 2 browsing tabs that you can easily swipe between.
The first is a “For you” tab which contains recommended content based on YOUR browsing habits (and a “Not Interested” option to curate, just like Tiktok).
The second tab is a “Following” tab which is strictly tweets from people you follow, in chronological order.
It’s pretty great now. Twitter used to be nothing but quasi activism and rage bait. Now it’s exactly what I envisioned it could’ve been, which is nice.
But stay away from it if you have the reddit mindset of “If I don’t agree with it, it should be banned”, because your feelings will get hurt.
It feels a lot like the internet 10-15 years ago, it’s great.
But I can freely talk about sports and shoot the shit without worrying about triggering an autoban for saying the “badspeak” term or phrase which is currently flavor of the month.
I said ‘in its default state’ because although you can try to configure it otherwise it always seems to gravitate back towards being shite. I commend you both for bothering, but I’d rather just use a service that is fundamentally less filled with rubbish. It always feels like fighting a losing battle.
If you haven’t used it recently, then you wouldn’t understand. But don’t use it if you have such strong feelings about it, it’s still a time sink.
I will say it’s one of the only services of its type that’s no longer a contrived echo chamber though. All opinions can be heard… But only if you seek them.
I’ve posted here before about how I refuse to use Twitter due to its toxicity (in the past), but its recent lack of toxicity has made it very fun and addicting lol
Now, if you’re a positive person who just enjoys tech and cat videos, that’s all you’ll see.
In the past, you could spend hours trying to curate Twitter to be good/positive, but you would still see nonsense.
The biggest improvement is the removal of the feature where completely unrelated content would be pushed into your timeline with a note at the top that said “Your follower [insert follower name] liked this”
So you were at the mercy of what the people you followed also engaged with. Now, you can follow someone who may be on the other end of the political spectrum (who may post cool tweets) - And while you don’t agree with their views on the election, you can still see their tweets without the “Pastor Bob liked this” type post appearing of QAnon bullsh*t.
There's actually a good reason. In short, a large portion of the apple silicon team ended up leaving a few years ago to could start a new company named Nuvia. Their goal was to produce high performance chips for the enterprise/server market, and they had some very aggressive performance targets [1].
Then, in early 2021 Qualcomm ended up acquiring Nuvia, and these new chips are the first showings of this acquisition. Naturally there's a lot of hype since said team represents a lot of the talent that made apple silicon so good in the first place.
Apple Silicon was great because Apple had invested huge amounts of resources for a decade on smartphone processors first, not because they had some kind of geniuses on the project.
It is no doubt promising that Qualcomm has brought in more talent, but it still takes time and effort to turn that into a best-in-class product. I’m not saying it’s impossible, though I’ll be skeptical of the hype until I see a real product.
> Apple Silicon was great because Apple had invested huge amounts of resources for a decade on smartphone processors first, not because they had some kind of geniuses on the project.
Right, and the guys who learned all the hard-won lessons along the way walked out the door to start a company, bringing along expert knowledge of Apple's designs and processes. And then Qualcomm bought them.
So on a surface level it seems implausible that QC could produce such a chip. But when you zoom out and go "oh, Qualcomm effectively bought Apple's senior chip engineers" it starts to make more sense.
It would be like if Qualcomm's top modem engineers started a company, which Apple bought. And then a couple years later Apple's long-running modem project mysteriously turned a corner and was ready to launch an exceptional modem. Like yeah, no kidding.
So yes, we need to see independent benchmarks and make sure it's not hype. But it's not so unbelievable that Apple's former top engineers could also produce a good a chip for another company. There's nothing magical about the Apple office--it's the engineers.
Engineering something as complex as a CPU is a long process regardless of how smart and experienced your engineers are. I mean, you can certainly speed it up with great talent, but there is still long and hard work to do with any difficult engineering challenge.
I’m not saying there’s something special about Apple other than the scale of their investment over a long period of time.
It’s the same deal for Qualcomm and their 5G modems. Apple no doubt has hired many talented engineers to make a custom 5G modem. But Qualcomm’s modem is still the best one around. It’s hard to catch up because Qualcomm has been investing heavily in that space for a very long time.
Again, that’s not to say Apple won’t ever catch up. Just that I wouldn’t expect that their first effort will be better than Qualcomm’s modems.
Nuvia has been working on this tech for years before being snapped up by Qualcomm. And before that, those same engineers had worked on Apple silicon for years. Why do you keep thinking this is an overnight thing?
To be fair it’s a bit of a myth that only mobile cares about efficiency and thermal management. It is definitely a factor for HPC and server too.
Apple scaled iPhone first designs up to the M* Ultra chips. Going from HPC to a mid wattage laptop is definitely serious work, but I don’t think it’s impossible. Especially with ARM.
The whole point of this thread is that those same Apple engineers made these Qualcomm chips.
Yes. Apple iterated over many, many years. Learning so much along the way about how to make performant, efficient ARM designs.
And then a bunch of the most important of those guys left to start their own company.
And then Qualcomm bought that company.
Y’all are acting like a few college kids from Stanford made Qualcomm a new CPU over their summer internship. “It takes longer than that to make a good CPU.” Yah no shit!
You are taking crazy pills. Making a high performance chip requires more than just having a bunch of talented and experienced engineers. Is it a necessary requirement? Sure! But it’s far from sufficient.
Apple brought on PA Semi and then slowly iterated on actually shipping hardware for years. They didn’t hire PA Semi and have a best-in-class product on the first go.
And those same guys who slowly iterated on shipping hardware for Apple for years are at Qualcomm now.
Are you saying it’s a requirement that these guys ship a crappy chip first? Why? They already know how to make good ones.
Can you tell me what more they need other than their talent and years of experience to make a good chip? Because if it’s just “I demand they make a bad chip now because they’ve changed logos on their corporate polos” I don’t think this conversation has anywhere to go.
Is the assumption here Apple has developed a business process for building best-in-class CPUs while treating its engineering workforce as fungible commodities? If so they've succeeded in doing what Intel has been trying to do for decades.
> Apple Silicon was great because Apple had invested huge amounts of resources for a decade on smartphone processors first, not because they had some kind of geniuses on the project.
Given that they were confident enough to leave and start their own company, I'm not sure this is true. Indeed I wouldn't discount the value of high talent density.
> I’ll be skeptical of the hype until I see a real product.
They have shown real hardware demos [1] to reviewers already, and the numbers look solid. Obviously there are no comparisons vs M3 yet, but it seems promising.
Not one or two Geniuses, but a really good team. They lost supposedly key members of that team which threw them off track. An organisation can recover from this, but it takes time and money. Not everyone likes to work under Apple-like working conditions towards Apple's goals.
Sure, but they’ve been behind the whole time. It’s not like they’ve been trading blows each generation with Apple.
If the premise is that Qualcomm hired a team of super talented engineers who can build a product that competes with Apple, then those engineers will still need time to develop a product.
Again, maybe their new processors will be everything they claim. It’s possible for this to happen. I’m just not willing to buy into the hype yet.
Surprised people don't know about this, as it has been common knowledge in the SD community [1] since october last year. Strictly speaking you don't even need cuda 11.8+ to get the speedup; it's sufficient to use cuDNN 8.6+, though you should use the newest versions for other reasons.
They took down the blogpost, but from what I remember the model is composite and consists of a text encoder as well as 3 "stages":
1. (11B) T5-XXL text encoder [1]
2. (4.3B) Stage 1 UNet
3. (1.3B) Stage 2 upscaler (64x64 -> 256x256)
4. (?B) Stage 3 upscaler (256x256 -> 1024x1024)
Resolution numbers could be off though. Also the third stage can apparently use the existing stable diffusion x4, or a new upscaler that they aren't releasing yet (ever?).
> Once these are quantized (I assume they can be)
Based on the success of LLaMA 4bit quantization, I believe the text encoder could be. As for the other modules, I'm not sure.
edit: the text encoder is 11B, not 4.5B as I initially wrote.
You'll be able to optimize it a lot to make it fit on small systems if you are willing to modify your workflow a bit: instead of 1 prompt -> 1 image _n_ times, do 1 prompt -> _n_ images 1 time -> _m_ times... For a given prompt, run it through the T5 model and store; you can do that in CPU RAM if you have to because you only need the embedding once so you don't need a GPU which can run T5-XXL naively. Then you can get a large batch of samples from #2; 64px is enough to preview; only once you pick some do you run through #3, and then from those through #4. Your peak VRAM should be 1 image in #2 or #4 and that can be quantized or pruned down to something that will fit on many GPUs.
> Interesting, though apparently the OPT175B model is 350GB:
Only in FP16. In the paper they use int4 quantization to reduce it to a quarter of that. In addition to the model weights, there's also a KV cache that takes up considerable amounts of memory, and they use int4 on that as well.
> I wonder what FlexGen is doing.. a naive guess is a mix of SSD and system memory.
That's correct, but other approaches have done this as well. What's "new" here seems to be the optimized data access pattern in combination with some other interesting techniques (prefetching, int4 quantization, CPU offload).
I want to emphasize how fascinating I find that the transform from 16 bit to a 4 bit quantization results in negligible performance loss. That's huge. Is the original FP16 not compressed?
The allowance for this more granular quantization seems to suggest the "bottleneck" is in some other aspect of the system, and maybe until that is addressed, a higher fidelity quantization does not improve performance.
Or maybe it's the relative values/ratio between weights that is important, and as long as the intended ratio between weights can be expressed, the exact precision of the weights themselves may not be important?
Found an interesting paper on this below. There's doubtless heavy research underway in this area
In my understanding, at a very high level and omitting many crucial details, the key is that when you have mainly largish matrix multiplications (as in transformers) well-behaved (mean zero uncorrelated random or so) quantization errors cancel out.
People do/did experiment with 1 or 2 bit compression of gradients/updates in the context of distributed training, but there it has been generally deemed useful to keep track of compression errors locally.
I had this happen to my XPS 15 when I was a student, and within 2 years of purchase the battery had degraded by 80%. If I recall correctly it would last about 20 minutes before abruptly turning off, making it completely useless as a laptop.
Absolutely maddening to hear that not only is this not an isolated issue, but it's also still not fixed (and apparently getting worse?).
There... is a torrent.