Not saying I believe in one side or the other, but from a standpoint of avoiding detection I think firmware hacking goes out the window.
A deep-pocketed attacker isn't going to risk flashing the firmware with a non-oem one on a brand new board leaving the factory. That probably gets quality inspected somehow later on anyway whereas a visual inspection is just a rubber stamp (IE: OK if the box isn't crushed or wet).
Not to mention a customer in the field who experiences problems is likely to report their firmware version to Supermicro support, whose poking around could expose the entire project.
There was an article recently about how hardware is "magic" and the IT world mostly takes it for granted. Putting an extra chip on the board but making it completely transparent to software debugging techniques is the best way to go. The board is almost certainly going to be flashed at least once and probably audited several times in it's lifespan by IT, but the hardware is never going to get more than some compressed air blown on it. Nobody repairs these things at the component level on a scale that matches how frequently firmware gets flashed or checked out.
Maybe for smaller companies, but Apple is very paranoid and AIUI does indeed inspect the hardware to make sure it hasn't been tampered with. I know less about Amazon in this regard but I would expect Amazon to do at least some level of hardware inspection to detect tampering as well.
They also do extensive code reviews, including of imported open source, yet their software is hardly bug free. Bugs can linger for years, often fixed only because someone stumbled upon odd behavior.
Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that....
The solution to these problems is to put critical code and critical secrets on discrete, simple SoCs where you actually have a chance of defending both hardware and software attacks. Apple and Amazon understand this because they already do it. The difficulty is building your software systems (firmware, kernel, etc) to make use of these secure elements, not to mention making them available for ad hoc application software. It's an extremely difficult integration problem, and even when you succeed you haven't.
For example, AFAIU Amazon's servers have secure elements to perform attestation of the box; it's utilized by their hypervisors to authenticate VMs for things like KMS. But it can't actually protect the data in the VM itself, such as the secrets obtained by dint of the attestation. It can't even prevent taking control of the hypervisor. All it does is help Amazon define a fixed security parameter--that you can't impersonate their hardware nodes on the network. That's extremely useful, but ultimately extremely limited.
Code reviews for bugs is very different than security reviews to look for malicious tampering. Apple may have bugs that linger for years, but I'm not aware of any documented case of someone managing to slip a backdoor into Apple software via an open source package.
> Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that....
Why not? X-raying every board, inspecting every single component, making sure it matches up with the documented specs and perhaps with a proven-good board... if you're replacing a component with a different one, or adding a component that wasn't there before (which is the case in this alleged attack), even if the component looks harmless and even if it's tiny, it can still be revealed by a detailed comparison of the board against specs. A component that's the size of a grain of rice is still a component that can be detected.
A deep-pocketed attacker isn't going to risk flashing the firmware with a non-oem one on a brand new board leaving the factory. That probably gets quality inspected somehow later on anyway whereas a visual inspection is just a rubber stamp (IE: OK if the box isn't crushed or wet).
Not to mention a customer in the field who experiences problems is likely to report their firmware version to Supermicro support, whose poking around could expose the entire project.
There was an article recently about how hardware is "magic" and the IT world mostly takes it for granted. Putting an extra chip on the board but making it completely transparent to software debugging techniques is the best way to go. The board is almost certainly going to be flashed at least once and probably audited several times in it's lifespan by IT, but the hardware is never going to get more than some compressed air blown on it. Nobody repairs these things at the component level on a scale that matches how frequently firmware gets flashed or checked out.