I would really encourage you to try your hand at at a monorepo. I manage a python monorepo in prod and dependency management is hell. Poetry has some newer features that I am looking at trying to implement, but the state of the ecosystem wrt big monorepos is horrible.
Poetry support for mono repos is really horrible and all the maintainers are saying is „it’s a tool indented to publish packages, not manage your dev environment“.
What is this non-package mode you are referring to?
I was talking about the maintainers not wanting to include features a lot of the userbase would like (like monorepo stuff), because they are saying their target audience is package authors, while in fact most of the users aren't.
I’ve been working on a thing [1] to make monorepo (“workspace”) builds easier, works well with Rye/uv. Doesn’t do anything during dev, just removes the need for hacks and scripts at build-time.
Would be curious if anyone thinks this is a useful direction, ultimately hope uv/hatch include something like this.
Bingo, this is what I was referring to in my comment. And yes, I assume it is quite a lot of labor. But I feel that lot of this perceived friction is from people trying to cut corners and avoid doing that labor, while still getting the benefits.
the overarching dream of computing in general is for a few people to do the labour and then for everyone to benefit through the magic of free replication.
if some smart and dedicated engineers can do the work to build a tool that lets everyone trivially manage a monorepo, that is certainly the best possible situation to end up in.
Package A depends on C 1.0 and B depends on C 2.0. How much work it is to get down to one version of C in your dependencies is up to how different 1.0 & 2.0 is, and how A & B use it. But if you want them resolved, it's up to you to do the engineering to A or B.
Correct. The solution is to modify packages A and or B, which is a high cost approach (hence why FAANGs could throw warm bodies at it, but most everyone else throws their hands up in exasperation).
So the canonical solution to this is to use Bazel. The reason I did not end up using Bazel is because it is a pita to manage and I didn't have enough time to allocate to this problem to implement the FAANG solution.
The shell scripts I wrote are a painful, but less painful than dealing with Bazel.
How is this personal? I have no idea who the person is... I'm commenting on something they wrote, not on something they are.
If a person in a home-cooking forum advises someone to use a butter knife to slice tomatoes, it's highly appropriate to tell them they are incompetent and shouldn't be advising anyone. This post is exactly the same: an absurd advise that can only be explained by either an honest mistake or incompetence.
it's highly appropriate to tell them they are incompetent
It's not on HN because it's a personal swipe and the HN guidelines ask you not to do those. It might work on your cooking forum but we know, empirically, that it doesn't work well here.
They didn’t say they were doing something goofy like that. They said they were running into problems at work. It could well be that they have a million-LOC repo that’s enormously complex with decades of legacy setup to consider. Presume that they’re competent at their job, and maybe what they’re trying to do isn’t trivially easy.
which means you need to come up with a bespoke dependency management solution. in either case, you're doing dependency management.
I encourage you to google "monorepo {python, poetry, pip}" and you're going to land on many multiple page blog posts describe arcane dependency management solutions.
The second option is traditionally what people mean by "monorepo" (multiple executables in a repo with a dependency tree). First one you'd call a "monolith" (single executable in a repo).
The second one you'd want to manage with a tool like Bazel, which will use whatever plugins are appropriate for the language (pip in Python's case).
It's a legitimate ambition to try and build a tool within a single language's ecosystem to manage the second option, but it's a really hard problem that Bazel (and others) have already solved well, so you might as well use them instead.
So you can do a somewhat hacky thing with poetry workspaces to get the first one to contain multiple binaries like i.e. https://github.com/bleu/balpy-poc
But yes, you're right -- Bazel is the most correct monorepo solution. We started with a combination of the second structure, poetry, and shell scripts but have since moved to nix.
Yea nix is an interesting angle of approach. It rhymes a lot with Bazel (reproducibility, explicit/declarative dependency tree, etc.), but nix approaches the problem from a different, lower layer of abstraction (by replacing the OS-level "pip" in apt, pacman, snap, brew, and the like).
I think, in the long run, something like nix will win out. It makes sense for your OS "package build system" to be the same as your project build system.
If your python project depends on some mildly obscure python lib, do you write your own nix "package" (or whatever nix chooses to call them) wrappers for each release you end up using? Any other tripwires in your experience so far?
Nix calls them "derivations", but yeah you're right again. Lots of weird language in this space.
Specifically, we use this nix project [1] which provides a nice translation layer between a poetry project and a nix derivation. It allows our devs to use poetry where they want (with local relative paths in the pyproject.toml) and ci/cd to have more granular control of deps at build time.
There are some corner cases. As you correctly guessed, some more obscure python libraries might require a few extra lines (i.e. to specify that this package needs setuptools etc). of code.
You should never use requirements.txt, nor pyproject.toml and this has nothing to do with whether you put multiple projects into the same repository or not. This is just an all-around bad idea.
But, anyways. In a monorepo project you would have nothing to do with requirements.txt or pyproject.toml because all dependencies are already there. There's no need to install anything from anywhere...
1. In monorepo you don't have dependencies. This is the whole point of having a monorepo: everything is included. You can build on airgapped system, no unexpected inputs into your builds, no network partitioning problems. This is the whole reason why people do that. Total control and stability. But you pay for it by having to maybe manage third-party code yourself. By having to use more space. Probably, you will need more infrastructure to side-step tools that cannot be made to not use network etc. Hope this also answers the question about external dependencies: they become internal dependencies.
2. Why you should never use requirements.txt: the tradition of using this approach comes from total misunderstanding of the goals of project deployment. This is surprising and upsetting, because this isn't a difficult concept. This is due to most developers not wanting to understand how infrastructure of their project works, and the willingness to settle on the first "solution" that "worked". The goal of deploying a project must be that byte-compiled Python code is placed in the platlib directory, accompanying data in the data directory and so on. The reliable way to accomplish this is to make a Python package and install it. Requirements.txt plays no role in this process, since package requirements need to be written into META file in the package info directory. Instead, the process that involves using requirements.txt typically ends up installing "something" that at the time of writing allowed the authors of the project to somehow make their code work, due to the combination of such factors as current working directory, some Python path files placed without their knowledge into platlib etc. This is a very fragile setup and projects designed in this way usually don't survive multiple years w/o updates that keep modifying requirements.txt to chase the latest changes in the ever changing environment.
3. pyproject.toml had more potential, in principle, but turned out to be a disaster. The idea behind this contraption was to organize configurations of multiple Python-related tools under one roof. But the chosen format was way too simplistic to realistically replace the configuration of other tools, and the configuration started to slide down two bad directions: either "string programming" (i.e. a technique where strings in the code develop special meaning and parsing), or delegation (i.e. pyproject.toml would contain a minimal code necessary to redirect to the real configuration). Second problem with pyproject.toml esp. when it comes to dependencies: there was no plan on how to solve this problem. No system. So, every tool that decided to support pyproject.toml decided to do it in a way that suits it. So, in the end, it's always better to just use the native configuration format of the tool that does the package building, instead of involving the middle-man: pyproject.toml. It always stands between the developer and their ability to debug the code, to fine tune the tool they need to run. It's the MS Windows registry all over again, but even more poorly executed.
----
So, what should you do instead? -- Well, there's a problem... there aren't any good Python tools :( And, like I mentioned before, the reason is even not the tools / their authors, it's that the underlying design of Python infrastructure is broken. So, you are bound to choose between the bad and the worse.
On the bright side: the problem, when you really understand what needs to be done is very simple. For any individual project you can solve all your deployment and packaging problems very easily writing in any language that can do infrastructure-related work. I've done this multiple times (as a result of frustration with Python's own tools) and had never a reason to regret it.
So this example is a single repo with many services? I don’t think OP was talking about that. Why would you have a single repo with different requirements? At that point they should be independent projects
> So this example is a single repo with many services?
Yes, that's the meaning of monorepo. If it was a single service it would be a monolith.
> Why would you have a single repo with different requirements?
Real world example: I implemented a client/server system using Python, and they have completely different requirements (even the Python versions are different). I still want to share code between client and server codebases, so a monorepo is the perfect choice.
You still have dependencies. But they are included in the repo.
What happens when you have 2 different apps in your monorepo but one uses an older version of Django and wasn't upgraded yet. The monorepo doesn't handle that automatically, you need tooling. It's not as black and white as you say.
In a company I worked for that used monorepo we had multiple versions of Linux (CentOS 6 variants) all at the same time. Trust me, something like different versions of Django is not a really big problem in comparison.
But, to answer your question in a more practical way: what are you going to do with these two versions of Django? Are you planning on running a single pre-fork server with two different Django application servers? Are there going to be two different pre-fork servers? Do both Django versions have to be loaded by the same Python interpreter? Or maybe they don't even need to be deployed on the same compute node?
Once you can answer questions like these, the solution becomes obvious. Most likely, what you want is something like two pre-fork servers proxying HTTP traffic into two separate instances of Django-based Python Web applications. So, in your deployment script, you create eg. two virtual environments and place two different Django packages into these two environments, together with associated code for the Web application.
There are, of course, ways to improve on that. Deploying while using Python packaging is, in general, wasteful. A better way is to merge all the packages you need to deploy your application into a single filesystem snapshot (removing all the info directories, and, potentially, all the source files, replacing them with bytecompiled ones, while also pruning all other irrelevant data from Python packages, s.a. readmes, test files etc.) Going even further, you can Cythonize your code, or even embed Python interpreter into your http server so that you deploy your application as a single binary. And there are plenty more of other options. The sky is the limit really.
This is maybe common, but this contradicts the definition of monorepo. You just use the word incorrectly. "Mono" means "one". If you pull packages from elsewhere, that stops being "mono". There's really no difference in this situation between your team publishing multiple packages from multiple repositories and then assembling them together for the purpose of deployment, or doing so, but with the third-party packages.
