I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.
I remember when Conda just appeared... I was so high no hopium... well, the word "hopium" didn't exist yet...
Anyways. Today I have to help scientists to deal with it. And... I didn't think it was possible to be worse than pip or other Python tools, but they convinced me it is. Conda is the worst Python program of note that I had ever dealt with. It's so spectacularly bad it's breathtaking. You can almost literally take a random piece of its source code and post it to user boards that make fun of bad code. When I have a bad day, I just open its code in a random file, and like that doctor who was happy running around the imaging room exclaiming "I'm so, so, so happy! I'm so unimaginably happy that I'm not this guy! (pointing at an X-ray in his hand)" I'm happy I'm not the author of this program. I would've just died of shame and depression if I was.
Simple in the sense that it's actually simple, the software you need can be installed with pip install with precompiled binaries for your platform when necessary it supports Python 3.something+, and all it's dependencies are either >= version or the version is >= x.y <= x+1.0
Then there's simple as in the software is actually incredibly useful but is an absolute nightmare of complicated dependency trees where only specific pinned minor versions work together, you need multiple incompatible compiler toolchains and distro packages, it only works if you have CUDA, precompiled binaries exist for some but not all and if you use the precompiled binaries then it changes the dependency story, if you want jupyter support that's a whole different thing AHHHHHHHHHHHHH
In that case some people with more time than sanity said fuck it we'll make it work and conda was born. For me it's a lifesaver when you want to use a piece of software but I wouldn't ever dare deploy production software with it without it being strongly isolated from everything else.
conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know).
poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nice but not something I've ever needed, but the buzz got out and it took over, people who have never typed "virtualenv" use poetry and they have no idea why.
Before conda, you needed a C and a fortran compiler, a BLAS and LAPACK installation, and various build tools, to install scipy. Scipy is one - 1 - dependency used in science and engineering. The pain used to be massive. Now, we’re complaining about the existence of 2 high quality (not perfect) alternatives, and pip (part of the official python distro) can install tensorflow etc, GPU libs included, with a oneliner.
> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know).
Essentially, yes, and justifiably so. Try installing science-y Python packages on Windows written in C. When conda was created in 2012 this meant installing Visual Studio 2005 (for Python 2.7) which was hard to find on Microsoft's own website even back then.
Conda is very nice for people who just want to write some python and not need to learn about the ecosystem or weird issues like some Visual Studio dependency not being installed to get scipy to compile on Windows or whatever. Like someone coming from R to Python. But aside from that situation I agree with you.
conda exists because it can install so much more stuff than pip without hassle, including things that aren't themselves Python packages, but which Python packages need. For example, if you ever tried to build a Python module that has some native code on Windows, you might appreciate this: https://anaconda.org/conda-forge/cxx-compiler.
so there's four responses here and 100% of them refer to a single Python package, Scipy, as the source of all the problems, where they had bad experiences over ten years ago with Python 2.
the Python package index now supports binary wheel files for all platforms and Scipy is there https://pypi.org/project/scipy/ with a few dozen distros.
Because often python is only one part of the puzzle, we also need a variety of third party libraries and maybe R as well (along with R packages). And we would rather not have to explain to another researcher how to install it all.
> (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know).
It started in a way similar to how Active Perl started. An attempt to build a reliable, curated and "batteries included" solution. (But it went downhill very quickly).
So, what were the problems:
* Most importantly, using Python on Windows is a pain (it still is, but less so). Many Python packages rely on native libraries which they don't provide. Even if they ship with some binary blob, that blob is often just a wrapper to the actual library. But, at the time Conda was created, even a binary blob was not an option. If you wanted to install stuff, you had to compile it. And, compiling on MS Windows is a world of pain. Especially, if you compile something for Python. Again, credit where it's due, Python sort of fixed some of these problems... but just a tiny bit. It still sucks a lot. So, conda is more of a MSYS2 really: it's a Linux user-space in Windows (firstly), and Python on top of that. But, it kind of also doesn't want to admit it, so, it still pretends it's Windows... but with a bunch of Linux-ey stuff...
* Secondly, pip was a joke until about a year ago. It was not really a package manager (it still isn't) and it couldn't properly solve dependencies (it sort of does today, but poorly because of source-distributions). Conda, when it worked, was at least somewhat consistent. (But usually, it just hangs). Also, conda is a package manager (however awful). I.e. with pip you can run install once, and then you can run a different install command again in the same environment, and god help you to make sure you have consistent dependencies between packages. Conda, in principle, should always keep your environment consistent. (But it hangs more often than not). Also, there's no such thing as installing from source with conda. If you want to use source: git clone, install conda-build and suffer through non-existent documentation and get help from non-existent community of people building conda packages.
* Conda provides a curated set of packages (a.k.a default channel). Anyone worth their salt ditches that channel the moment they install conda and installs everything from conda-forge (cuz its got more stuffs!) So, they tried curated. Didn't work.
---
In a sense, I think that what happened is that conda was too ambitious of a project, using wrong technology to implement it. It also didn't have good enough planning for the kind of ambition they had. So, every time they came to solve a real problem, they didn't have time, human resources and system resources to solve it well. They've accumulated technical debt at close-to-light speed and ended up being a huge mess nobody knows how to deal with.
Some of the obvious mistakes would be: creating too many releases of conflicting packages. They should've worked on some sort of LTS solution, where they release a set of packages with very permissive dependencies of very few versions that had been proven to work well together. Instead it's very common for conda packages to be very peculiar (and without real need to do so) about the versions of their dependencies.
Conda people didn't build good CI. They often release absolutely broken packages, and only discover it retrospectively from community input. (Tensorflow would be a great example). This creates a lot of problems with automatic updates.
Conda gave in to community pressure and decided to build integration with pip. It doesn't work well, and it's not possible to make it work well, but they added this and a lot of people instantly created dependencies on this feature.
Conda picked a bone with some of the infra tools outside of Python ecosystem. In particular with CMake. This just adds an extra aspect of pain trying to build conda packages / work with native libraries / wrappers. It might seem like a minor thing, but it prevented a lot of people who might otherwise release packages both for conda and PyPI from doing so. Often package that is released to conda is ported by someone who's not the package author. It also means that sometimes the names of packages differ between conda and PyPI for the same package.
----
NB. In terms of amount of commands you need to type when working with conda vs with PyPI tools is not noticeably different. Conda is, perhaps, more organized, but is also more buggy due to trying to integrate with various shells in special ways, and failing quite a bit.
Any program with real world application solves some problem. That's not the point. The point is in how does it do it. And when it comes to conda... it's like MS Outlook on steroids. I cannot really think about a better way to describe it. It's like a movie that can be so bad that it's actually good.
I'm the opposite. We have to maintain a lot of different environments for different projects, and with conda things "just work" (esp now with better PyCharm integration). Venv is much more of a hassle.
I have like seven pythons on my machine, and I use virtualenv all over, there's no issue, what's the issue? I have to type the path of a specific interpreter when i make a virtualenv, is that the problem? I bet that's the problem
> With plain venv it’s hard to maintain multiple different Python versions on the same machine
Ironically, given the usual software dev experience on Windows vs. Unixy systems, this is not a problem with the standard install on Windows with the py launcher.
It's not ironic at all; traditional Linux package management is actualy really bad. On a Linux system you can basically only have one version of anything installed at the same time.
I have multiple Python versions on both my Macbook M1 and my Linux laptop, with several dozen venvs. I honestly don't see why others are having so many issues.
Presumably they're not managed by your system package manager? Or you have one of the rare and relatively new distributions (certainly not typical Linux) that can handle that smoothly, like NixOS?
Is it? I just install different version with brew, and choose which python3.X to create a venv with. The ML packages issue is much more of an issue, but now that the ARM transition is well underway more and more packages work via normal pip.
> With plain venv it’s hard to maintain multiple different Python versions on the same machine;
plain venv never advertised itself as a solution to this problem... I don't like this tool, but, sorry to say so, you are barking on the wrong tree.
Also, it's not hard to maintain different versions of Python on the same machine without conda. I can literally do this with my eyes closed w/o touching the mouse: it boils down to:
cd ~/src/cpython
git checkout 3.8
git fetch --all
git reset --hard origin/3.8
git clean -Xdf
./configure --with-ensurepip=install --with-system-ffi=yes
make
sudo make altinstall
Sorry. I've done this from memory and I kept one eye open. So, I kinda lied. But not really.
> Just a python -m venv per project, a requirements.txt, and you will basically never have issues.
As long as you always remember to run exactly the right two commands every time you work on any project and never run the wrong ones, or run project A's commands in project B's terminal. There's no undo BTW, if you ever do that you've permanently screwed up your setup for project B and there's no way to get back to what you had before (your best bet is destroying and recreating project B's venv, but that will probably leave you with different versions of dependencies installed from what you had before).
(And as others have said, that doesn't cover multiple versions of Python, or native dependencies. But even if your project is pure python with only pure python dependencies that work on all versions, venv+pip is very easy to mess up and impossible to fix when you do)
Numpy and Scipy are good reasons. Unfortunately Scipy does not even compile on FreeBSD lately, and I have opened three issues about it against Scipy and Pythran (and the fix was with xsimd).
