Antithesis | Distributed Systems Breaker Extraordinaire | Full-time | SF or London or DC
---
Have you always wanted to channel your inner Kyle Kingsbury? We are looking to hire somebody with a deep understanding of distributed systems who can bring open source or customer software under test in our platform and find fun issues in it.
Preferred locations: SF, London, DC. Remote is a possibility for this role if you're really good (though be aware that we're a very in-person company culturally speaking, so it might be annoying for you).
To apply: email 'will' at our company domain with a description of something interesting that you've worked on.
I am one of the people who was on the panel. As always, it's a very lossy process when a journalist is summarizing another journalist summarizing a 90 minute discussion. Happy to expand in the comments here on any of the issues that got brought up.
You stated that “the conversation about where the technology is going and what we’re going to do with it is happening among people who do not care … what any Christian church has to say on the topic.”
As a Catholic myself, I wonder what the church should say/do with regards to technology like this and would like to know where you stand on it. Personally I think that further public discussions need to be held on the morality of potential implementations of AI (I’m thinking ahead by decades once LLMs can really make a dent in the workforce) but question the effectiveness of the Church participating in discussions like that.
It’s not quite a fair comparison, since an RL agent is trying to learn a policy that wins fair and square, while a fuzzer is able to take back moves. But if you’re working in a domain (like anything that can be simulated) where “time travel” is possible, you’d have to be crazy not to use it!
The longer you run it, the cleaner the run gets. But Metroid is a very compute-intensive game to fuzz, and we were already nearing the limits of what BigQuery could do for us with that run.
Yes, this is a really fun idea and something that we want to do. Though these days we’re setting our sights higher than Nintendo…
A funny story though: a regular conference gimmick we have is “Man vs. Machine” where we have attendees race our fuzzer to the end of Mario level 1-1. We did this at the final year of Strange Loop, and the fuzzer was winning handily until not one, not two, but three different professional speedrunners walked by and destroyed us.
This seems like a cool company and I don't want to nitpick too much, but gamers have no respect for history:
Castlevania... [so] called because it is a Metroidvania game set in a Castle.
Ouch - this is precisely backwards. Metroidvanias are named after Metroid and Castlevania because those series practically defined the genre.
Also a bit frustrating because the first Castlevania itself isn't actually a metroidvania, it's a more conventional action-platformer. Castlevania II has non-linear exploration, lots of items to collect, and puzzle-solving, all like Metroid. So it's not too surprising Antithesis had to do a lot of work for adapting their system to Metroid - but I wonder if this work means it now can handle Castlevania II without much extra development.
There have historically been two giant adoption challenges for DST.
(1) Previously, you had to build your entire system around one of the simulation frameworks (and then not take any dependencies).
(2) It’s way too easy to fool yourself with weak search/input generation, which makes all your tests look green when actually you aren’t testing anything nontrivial.
As you say, Antithesis is trying to solve both of these problems, but they are very challenging.
I don’t know of anybody else who has a reliable way of retrofitting determinism onto arbitrary software. Facebook’s Hermit project tried to do this with a deterministic Linux userspace, but is abandoned. (We actually tried the same thing before we wrote our hypervisor, but found it didn’t work well).
A deterministic computer is a generically useful technology primitive beyond just testing. I’m sure somebody else will create one someday, or we will open-source ours.
DST was i̶n̶v̶e̶n̶t̶e̶d̶ popularized at FoundationDB a little over a decade ago, and has been quietly gathering steam ever since. If you’re interested in the technique, the FDB paper has some good info in section 4 and section 6.2: https://www.foundationdb.org/files/fdb-paper.pdf
(Disclosure: I am an author.)
I also gave a talk overview of it at Strange Loop back in 2015, but don’t have the youtube link handy.
If you’re interested in trying DST, we have a new company that aims to make it a much easier lift, especially for existing projects that can’t be retrofitted onto one of the many simulation frameworks: https://antithesis.com
Happy to answer any questions about the approach, here or over email.
Hey Will. I'm a huge fan of the work you all are doing, and of FoundationDB, but I don't believe it's accurate that DST was invented at FoundationDB (or, maybe it was, but was also used in other places around the same time or before).
For example, the first implementations of AWS's internal lock service (Alf) used DST as a key part of the testing strategy, sometime around 2009. Al Vermeulen was influential in introducing it at AWS, and I believe it built on some things he'd worked on before.
Still, Anithesis is super cool, and I really admire how you all are changing the conversation around systems correctness. So this is a minor point.
Also a huge proponent of Antithesis and their current work, but there definitely were some notable precedents at or around that time e.g. MODIST from 2009 (https://www.usenix.org/legacy/event/nsdi09/tech/full_papers/...), which similarly tried to build a "model checker for distributed systems".
As another interesting historical side note, I have wondered about the similarities between Antithesis and "Corensic", a startup spun out of UW around a similar time period (circa 2009). Apparently, they "built a hypervisor that could on-demand turn a guest operating system into deterministic mode" (see "Deterministic Multiprocessing" at
https://homes.cs.washington.edu/~oskin/). My impression is that their product was not a significant commercial success, and the company was acquired by F5 Networks in 2012 (https://comotion.uw.edu/startups/corensic/).
Overall, I don't over-index on novelty, and think it is generally good for ideas to be recycled/revived/re-explored, with updated, modern perspectives. I believe that most rigorous systems designers/engineers likely converge to similar ideas (model checking, value of determinism, etc.) after dealing with these types of complex systems for long enough. But, it is nevertheless interesting to trace the historical developments.
Corensic was impressive tech. I actually debriefed with one of their founders years ago. IIRC, their product was focused on finding single-process concurrency bugs.
Deterministic hypervisors are by no means new. Somebody once told me that VMWare used to support a deterministic emulation mode (mostly used for internal debugging). Apparently they lost the capability some time ago.
Hi Marc, thank you for the correction! We started doing it around 2010, and were not aware of any prior art. But I am not surprised to hear that others had the idea before us. I will give Al credit in the future.
