It feels like it is one of the reasons experienced devs are ditching Python for production systems. Besides horrendous performance and lousy semantics. The cost of setting up, maintaining the environment and onboarding people is just not worth it.
Excuse my ignorance, but aren’t Virtual Environments something you setup once per project? Why would that be a dealbreaker? How is it any more difficult than putting everything in a docker container like all the cool kids are doing these days?
* Local state is changing such as brew updates or new dependencies are added.
* External state is changing such as project contributions
So its not a one-off unless the project and dev environment is static. The real problem is different tooling doing different amounts of hand holding and automation. Your editor may configure some things automatically, brew may configure some things automatically, and so a set of instructions for setup or problem fixing could be voided without the end user knowing. So now you're off on a adventure of unknown duration wading through internet forums trying to find the resolution that works for you.
Ironically using Docker to isolate the environmental changes is a approach some people use to avoid some of this esoteric crap.
I'd even dare to say that Docker _is_ the answer to the Python's packaging problems, and might have never taken off without that "killer app" that is the Python packaging sh*tshow.
It's only a workaround unless the python env in the resulting docker is reproducible for all practical purposes. Otherwise the generated images become precious artifacts you can't loose - there is a risk you gonna get problems if you re-build the image now instead of reusing the working one built X years/months ago.
Docker is not the answer to any packaging problems because it's not a packaging tool. I have no idea how people don't understand this... oh wait! I'm talking to Python programmers!
But... I'm not going to hold you in the dark: the reason and the major drive to have a packaging system is that you can define dependencies between packages s.t. users installing package can coordinate and install the stuff they need. Docker simply doesn't do that. You get images. They are completely independent. Whether any two images will work with each other is anyone's guess.
Your post reminds me of the saying with the faster horses.
Docker doesn't solve Python's packaging problems by being a better packaging system. It solves it by sidestepping the issue of users installing packages themselves, and shipping the disk of the one machine where it once worked.
It's a bad linker.
(not a fan of the condescending tone in your post, btw)
Lol. It doesn't. How are you going to create Docker images? Run more Dockers inside Docker? And then Dockers all the way down? You still need to install the packages... just while creating an image. Makes absolutely no difference whether you do it inside or outside, you still need something that can install and coordinate packages. Docker is simply irrelevant to this problem.
Once. On a build server. Or a dev's workstation who had the time to fiddle with all the breakage that is python envs. This absolutely does sidestep the issue of distributing Python packages to your users.
We're talking about different problems here. You are talking about library dependencies for devs (and I understand that I'm a bit off topic for this thread)
I didn't ask where, I asked how. How will you know what packages do you need to install? Are you going to go on pypi.org, download them one by one, unzip, look for requirements, then go to pypi.org again, download more, and so on, until you have enough packages and have figured out all the requirements? If so, I have very bad news for you: your single installation might take weeks, possibly months, or might in fact be so slow it will never end as new packages will be published faster than you can identify the version you need.
> This absolutely does sidestep the issue of distributing Python packages to your users.
No it doesn't... OMG. You cannot be serious? Do you really search and solve dependency problems w/o using any dedicated software? Like in your head? Or with pen and paper? For any non-trivial project the number of combinations you have to try before you find the one that works will be astronomical...
> We're talking about different problems here.
Nah. You just don't understand the problem. It's similar, but not the same.
You have a very abrasive style of dialog, combined with an unwillingness to consider alternative POVs in your discussion, bordering on trolling. I like discussion, but I don't enjoy this. With that, I will disengage now.
You don't have an alternative point of view. You don't have a point of view, because, again, you don't understand something very basic about this subject.
Python has always struggled with distribution maintainers that just don’t get it.
Python barely survived becoming the default scripting language on Red Hat and other Linux distros which was a major obstruction to the Python 3 transition. If the new cohort of pandas and scikit-learn users had not been such a force of nature we”d be talking today about Python the way we do about Perl.
Not installing venv is a serious crime on the part of Debian as beginners don’t need any more excuse to trash their system with site-local directories and other wrong answers for how to manage packages in Python.
People just want to get things done. If they can get away with two commands instead of six they will, which is why I think poetry has gotten more adoption.
The word package is overloaded. There is a Debian package and there is a Python package.
If somebody wanted to package a modern Python application as Debian package they’d have to make a Debian package that contains several Python packages maybe as a venv or executable wheel, it is a solvable problem but a bit like comparing tic tac toe to 12d chess with supersymmetry in terms of the ambition of Linux distros if not the difficulty.
If you installed python packages as debs without any isolation you'd never be able to install two applications that depended on different versions of the same Python package.
The best thing about Java is that it is so xenophobic that nobody expects to have a working Java runtime installed with the OS so you never have to deal with a broken runtime installed with the OS. JAR files are remarkably self-sufficient so "write once one anywhere" is way closer to the truth than it is on "DLL Hell" platforms like Windows and Linux.
It takes more work to create and push a Debian package. Yes. But once it's done, it's DONE, and you never talk about it again. This whole article and thread can disappear, which is a MASSIVE time savings. The "DLL hell" you're referring ot exist on platforms that don't have package management, which notably does not apply to Linux, in general.
The funny part about Java is that "Enterprise Java" used to depend a lot more on things being in the base application server, and because that was, as usual, complete hell, it now went the complete other way and microservices will often "just bundle the entire application server" in the uberjar as a result.
Because people are lazy and the right thing must be easier then the wrong thing. The main project in Python now (going well) is removing the footguns, and
pip install —-user
Is one of them. If users don’t have venv they will trash their Python installation almost right away and probably join the many people who’ve left Python.
> Isn’t it considered best practice not to use brew to install Python for exactly this reason?
The only time I've ever had a big mess with broken Python installs was after using brew on Linux - luckily killing off brew brought the system Python back fine. I'll grudgingly put up with brew on a Mac out of necessity, but keep it away from Linux.
Not that I would truly recommend to use brew on linux as there are better alternatives - docker images, distrobox with a rolling distro for newer python, or LTS distro to keep an older version, pyenv etc, but..
If you've had a problem with brew, it was most certainly caused because you sourced it in your .profile, .bash_profile or .zprofile. Never do that.
It's what brew's homepage recommends, but it can break a LOT of things and not just python. Brew should be in the $PATH only when you need it, from your current instance of interactive shell used for development, and not login-wide. The brew version of various pieces of software should not ever be able to take over what your system has or very unpredictable things will happen.
I avoid brew whenever possible, partially because it does weird stuff just to avoid installing things as root. MacPorts is nice but doesn't always have the port you want. Something like Python with a convenient .pkg download directly from python.org makes the choice obvious.
That said, some Python packages rely on native code that you might find yourself brew-installing. That can be a nightmare.
I prefer to not let my editor do any package management at all, unless everything is put into config files, which are standard. For example in pyproject.toml for poetry. Simply putting a Makefile can make things comfortable enough. No need for editor specific magic or depending on buggy plugins, that do things right only in 90% of the cases.
> but aren’t Virtual Environments something you setup once per project?
My typical count is 2-3 per project per machine I work on (could be anywhere from one to one hundred). Then there's a different number of people who need to set up these environments on different machines too (and sometimes require my support).
So, the answer is: who knows?
> putting everything in a docker container
I think, you meant image, not container, but that's an easy mistake to make. And the answer is: both options are terrible. It's a choose your poison kind of thing.
> cool kids
My impression from working with the cool kids is that our industry selects for particular taste in t-shirts rather than knowledge or experience. I'm afraid that the correlation might swing into the negative, if we take either knowledge or experience vs coolness.
---
Most importantly: venv is not the problem. It's a bad fix for a problem. Bad as in it doesn't fix the problem actually, it pretends to. I mean, maybe it covers some 90% of the problem -- who knows, I didn't count. So, it kinda works for a lot of people. But, honestly, I'd prefer that the problem was fixed s.t. it doesn't require venv. It's kind of like discussing various dental prosthetics options: it's better to just have healthy teeth.
> The cost of setting up, maintaining the environment and onboarding people is just not worth it.
I have yet to come across a situation where I need a virtual environment at all. A lot of projects use it, but then lazy me just runs git clone && python3 clone/main.py and it works just fine, sometimes after an apt install python3-somedependency or two.
It always seemed weird to me to depend on such a specific version ("package foo can't be older than 7 months or newer than 4 months"), how even does one manage to use such obscure features that they were removed and so you need an older-than version?
And then there's the people in this thread (seemingly a majority when judging by top level comment loudness) that have trouble with virtual environments and so add another layer on top to manage that virtual environment thing. The heck. Please explain
Do you deal with the scientific libs? I remember that whole MatPlotLib/Scipy/Pandas/Jupyter/whatever stack having weird requirements on dependencies, with Py2 vs 3 confusion added to the mix.
I think the problems have been resolved. For my last few installations of Python, I've just used pip install for those packages, without any issues. Linux and Windows, can't comment about Mac. And the important libraries are all in Py3 now.
Yeah I'm willing to bet it's gotten easier lately, perhaps because things have settled down. Then again Mac is always its own special case with Py libs.
I've used matplotlib and pandas fairly recently because applicants used it in their code, don't remember any specific problems with that. Well, with the dependency installation that is. The applicant loading all data into RAM so that Pandas can operate on it was an issue. (The whole point was that the data was too big to reasonably load into RAM, and would constantly grow as more sensor readings come in, and the questions are such that you can easily stream it and keep a few KB of state in a dictionary to find the answers needed... alas.)
I do remember python2-only being a problem back in the day, but this was solved... hmm, maybe in 2017 somewhen? At least for the packages I used then that had py3 issues before, like sklearn, numpy, and scapy come to mind. I think it more or less coincided with Debian deciding the next release was not going to ship Python 2. Somehow that made everyone 2to3, fix a few remaining bugs, release, done. I'm too young (I'm 30) to really have done much with Python 2 so I didn't have this legacy in my own code (besides a few early files), I've ~always just tried to find and install Python 3 versions of everything.
Experienced Python dev checking in here. I use neither containers nor virtual environments. Honestly not sure what all the fuss is about.
Maybe everyone has come to think we need layers on layers on layers because management tools (like venv) are blogged and talked about, whereas it's a bit dull to write/talk about nothing (i.e. not using/needing such tools)? I genuinely wonder
