I had to move my system to a new SSD with more space, mostly because the great Python venv system makes it that each project takes up between 5Go and 30Go of disk space, so having 224Go was not enough anymore.
Nvidia driver didn't install properly so second reboot was to a black screen. Thankfully after some fiddling with packages it worked again.
Then the old ways I loaded up ssh-agent didn't work anymore, so it took a few logouts to find a new one.
I tried to recompile my CV, but somehow Latex dependencies all changed names, so I had to track down missing packages. Then it still would not compile because the names of the fonts had changed. Configuring the font line was frustrating as xelatex and pdflatex did not expect the same line, and both crashed on a different error that did not lead to an easy solution.
When I tried some game on Steam, it just crashed without an explanation. After checking ways to boot it from the CLI -- which is unsupported but search results lead to partial solutions to that problem anyway -- I found that it was possible to add a DEBUGGING argument to the proton command. Searching the error lead to simply find that some lib was missing (actually the challenge was in finding the proper error to search for). I'm the most amazed at this one since this is not in any way "developer software", but a standard user one, for which at least 1 hour of pain would have been removed by an optional dependency.
I'm cutting down the list but it took days to solve every single broken thing.
By coincidence I watched "how to prevent the collapse of civilization" by Johnathan Blow. His point that nobody is surprised anymore that everything breaks and is bugged, it has become the baseline expectation.
In my Fortran IV class fifty years ago, and it was not unusual for the Burroughs mainframe to be down when one showed up with punch cards or showed up to punch them.
Modern cars are vastly more reliable than the ones I grew up driving. Modern tires last a lot longer. It is true that sixty years ago one could pull tubes out of the TV and test them at the hardware store's plugboard; but it is also true that new TVs without such troubleshooting options are much more reliable. I think that we got our current TV because my wife wanted to watch the 2008 presidential debates. It still works.
As other mentioned, anything AI (Stable Diffusion for instance is 5.86, ComfyUI 5.19, then there is Whisper, OpenLyrics, LLM), ... this is not counting the model files of course.
Some client projects are packaged in a way that bring in a lot of unneeded dependencies. They could be repackaged to be 'lite' but no one thinks this is worth the time investment and is going to pay for it.
In almost all cases venv are needed because the dependencies are brittle and changing the python version or some package version break the program.
Considering that those sizes don't even include the models, what is actually taking up several gigabytes? How is it so much code? Or is it more than code?
I know it includes dependencies, but I'm still baffled.
Like the other comment mentions my venv also regularly reaches 5Gb thanks to deep learning libraries.
The Nvidia packages alone take 2.8G and torch another 1.4G. Numpy, transformers, matplotlib and two dependencies named sympy and triton add another 500 MB, and with that alone we're already at 4.7G.
I dunno about the Blow part. IME since the late 80s computing was mostly wrestling with bugs, crashes, and issues you had to solve mostly on your own, or just live with (t. Guru Meditation vet), and it was only worse during the 9x era. The relatively recent "everything works and is stable" era was more of an anomaly than how it always was.
I think it's a growing experience to some degree anyway. Suffering through Windows 98SE taught me some pretty useful practical skills and expectations that are still applicable in some capacity even today.
Did you want to reinstall or did you not know better? For the latter case, dd (or cp actually, you rarely need dd!) and resize2fs (similar tools exist for other filesystems) have served me well to move to new storage with minimal effort. Copy the whole disk, resize the partition, resize the FS, set new boot drive, boot from new drive.
I thought it would be a good opportunity to improve some of my setup, in particular because I migrated my work config that covered way more cases. Also since I had all those setup scripts that I wrote to setup servers (git, python, postgresql, nginx, ...) I expected it would barely take more time (this part went fine actually!). In hindsight I regret not going the dd way which I had initially planned, even if the new setup is a bit cleaner.
I have been suffering from this venv problem as well. Anyone know if there is a programme or effort to dedup dependencies across projects or something? Feels like it is probably doable.
You can create a virtual environment that you don't source/activate (for your projects), then you can use that to `pip install` the libraries you want to share and add that to your `PYTHONPATH`.
Since your `PYTHONPATH` searches in order your project virtual environment will override any other libraries you have in your path, say with different version.
Will pip also see that and not install what’s installed there? This is most useful in AI tools, but they tend to have their own “setup” process that pip-installs everything regardless. If not, it’s no use.
I'm cutting down the list but it took days to solve every single broken thing.
By coincidence I watched "how to prevent the collapse of civilization" by Johnathan Blow. His point that nobody is surprised anymore that everything breaks and is bugged, it has become the baseline expectation.