This is super interesting, as I maintain a 1M commits / 10GB size repo at work, and I'm researching ways to have it cloned by the users faster. Basically for now I do a very similar thing manually, storing a "seed" repo in S3 and having a custom script to fetch from S3 instead of doing `git clone`. (It's faster than cloning from GitHub, as apart from not having to enumerate millions of objects, S3 doesn't throttle the download, while GH seem to throttle at 16MiB/s.)
Semi-related: I always wondered but never got time to dig into what exactly are the contents of the exchange between server and client; I sometimes notice that when creating a new branch off
main (still talking the 1M commits repo), with just one new tiny commit, the amount of data the client sends is way bigger than I expected (tens of MBs). I always assumed the client somehow established with the server that it has a certain sha, and only uploads missing commit, but it seems it's not exactly the case when creating a new branch.
Funny you say this. At my last job I managed a 1.5TB perforce depot with hundreds of thousands of files and had the problem of “how can we speed up CI”. We were on AWS, so I synced the repo, created an ebs snapshot and used that to make a volume, with the intention of reusing it (as we could shove build intermediates in there too.
It was faster to just sync the workspace over the internet than it was to create the volume from the snapshot, and a clean build was quicker from the just sync’ed workspace than the snapshotted one, presumably to do with however EBS volumes work internally.
We just moved our build machines to the same VPC as the server and our download speeds were no longer an issue.
When you create an EBS volume from a snapshot, the content is streamed in from S3 on a pull-through basis. You can enable FSR which creates the EBS volume with all the data up front, but it is an extra cost option.
Yeah, this is exactly my point. Despite provisioning (and paying for) io1 ssd’s it doesn’t matter because you’re still pulling through on demand over a network connection to access it.
It was faster to just not do any of this. At my current job we pay $200/mo for a single bare metal server, and our CI is about 50% quicker than it was for 20% of the price.
Hmm I don't know that making a new volume from a snap should fundamentally be faster than what a P4 sync could do. You're still paying for a full copy.
You could have possibly had existing volumes with mostly up to date workspaces. Then you're just paying for the attach time and the sync delta.
> I don't know that making a new volume from a snap should fundamentally be faster than what a P4 sync could do. You're still paying for a full copy.
My experience with running a c++ build farm in the cloud is that in theory all of this is true but in practice it costs an absolute fortune, and is painfully slow. At the end of the day it doesn’t matter if you’ve provisioned io1 storage; you’re still pulling it across something that vaguely resembles a SAN, and that most of the operations that AWS perform are not as quick as you think they are. It took about 6 minutes to boot a windows ec2 instance, for example. Our incremental build was actually quicker than that, so we spent more time waiting for the instance to start up and attach to our volume cache than we did actually running CI. The cost of the machines was expensive that we couldn’t justify keeping them running all day.
> You could have possibly had existing volumes with mostly up to date workspaces.
This is what we did for incremental builds. The problem was when you want an extra instance that volume needs to be created. We also saw roughly a 5x difference in speed (IIRC, this was 2021 when I set this up) between a noop build on a mounted volume and a noop build that we had just performed the build on.
There’s a service called p4vfs [0] which does this for p4. The problem we had with this at the time was that unfortunately our build tool scanned everything (which was slow in and of itself) but that caused p4vfs to pull the file anyway. So it didn’t actually help.
This is fascinating, I didn't know they did this. This is actually not using the built in functionality that Git has, they use a shell script that does basically the same thing rather than just advertising the bundle refs.
However, the shell script they use doesn't have the bug that I submitted a patch to address - it should have all the refs that were bundled.
git.kernel.org hits one of the frontends based on geographic location. I'm not sure how often it's discussed, but see [1], and also `dig git.kernel.org`.
Have you looked into Scalar? It's built into MSFT git and designed to deal with repos that are much larger internally.
microsoft/git is focused on addressing these performance woes and making the monorepo developer experience first-class. The Scalar CLI packages all of these recommendations into a simple set of commands.
scalar and msft git (whose many features made it into mainline by now) addresses mostly things like improving local speed by enabling filesystem caching etc.
It doesn't address the issue of "how to clone entire 10GB with full history faster". (Although it facilitates sparse checkouts, which can be beneficial for "multi-repos" where it makes sense to only clone a part of repo, like in old good svn.)
To try this feature out, you could have the server advertise a bundle ref file made with `git bundle create [bundle-file] --branches` that is hosted on a server within your network - it _should_ make a pretty big difference in local clone times.
I can't imagine you haven't looked at this, but I'm curious: Do shallow clones help at all, or if not what was the problem with them? I'm willing to believe that there are usecases that actually use 1M commits of history, but I'd be interested to hear what they are.
Maybe I was doing something wrong, but I had a very bad experience with - tbh don't remember, either blobless or treeless clone - when I evaluated it on a huge fast-moving monorepo (150k files, 100s of merges per day).
I cloned the repo, then was doing occasional `git fetch origin main` to keep main fresh - so far so good. At some point I wanted to `git rebase origin/main` a very outdated branch, and this made git want to fetch all the missing objects, serially one by one, which was taking extremely long compared to `git fetch` on a normal repo.
I did not find a way to to convert the repo back to "normal" full checkout and get all missing objects reasonably fast. The only way I observed happening was git enumerating / checking / fetching missing objects one by one, which in case of 1000s of missing objects takes so long that it becomes impractical.
For rebasing `--reapply-cherry-picks` will avoid the annoying fetching you saw. `git backfill` is great for fetching the history of a file before running `git blame` on that file. I'm not sure how much it will help with detecting upstream cherry-picks.
Oh, interesting! Tbh I don't fully understand what "--reapply-cherry-picks" really does, because the docs are very concise and hand-wavy, and _why_ it doesn't need the fetches? Why it is not the default?
Indeed, `git ls-remote` produces 14MB output; interestingly, 12MB of it are `refs/pull/<n>/head` as it lists all PRs (including closed ones), and the repo has had ~200,000 PRs already.
It seems like large GitHub repos get an ever-growing penalty for GitHub exposing `refs/pull/...` refs then, which is not great.
I will do some further digging and perhaps reach out to GitHub support. That's been very helpful, thanks Scott!
Have you switched already to the "new" git protocol version 2? [1]
> An immediate benefit of the new protocol is that it enables reference filtering on the server-side, this can reduce the number of bytes required to fulfill operations like git fetch on large repositories.
Semi-related: I always wondered but never got time to dig into what exactly are the contents of the exchange between server and client; I sometimes notice that when creating a new branch off main (still talking the 1M commits repo), with just one new tiny commit, the amount of data the client sends is way bigger than I expected (tens of MBs). I always assumed the client somehow established with the server that it has a certain sha, and only uploads missing commit, but it seems it's not exactly the case when creating a new branch.