Oh I see, so the problem is that the merge commit is not the same as the one that the test runs for if there are any intervening commits between them. Interesting, I didn't quite realize that! (Though it seems obvious in hindsight, since a new push doesn't make all past PRs' tests re-run.) Thanks for explaining that! So it seems it's something between what they do and what they were trying to avoid. GitHub should really put a button to re-run tests on merges that are out of date...