- Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers.
- SystemTap, a scripting language and tool diagnose of a performance or functional problem.
- Fail, gofail for go and fail-rs for Rust
- Namazu: a programmable fuzzy scheduler to test a distributed system.
We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage
tl;dr: Chaos Engineering
Some links: https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engin...
https://www.pingcap.com/blog/chaos-practice-in-tidb/
Regarding the Fault Injection tools we are using:
- Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers.
- SystemTap, a scripting language and tool diagnose of a performance or functional problem.
- Fail, gofail for go and fail-rs for Rust
- Namazu: a programmable fuzzy scheduler to test a distributed system.
We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage