Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The NTSB's original report has more detail on how the SD Card was encrypted and how the NTSB managed to decrypt it:

https://data.ntsb.gov/Docket/Document/docBLOB?ID=18741602&Fi...



The System on Module board is an Inforce 6601 SOM. [0]

It uses a Qualcomm Snapdragon 820 and they provide prebuilt Ubuntu Linaro distros for it, preconfigured for the board.

The camera manufacturer likely just tossed it straight in as configured and thus didn't know how the full disk encryption was setup.

This whole camera design looks like one of those 'we gave this project to some undergrad engineering students who've never designed a commercial product before and had no price target and thus it has a whole damn embedded linux system inside it for merely taking some HD video and stills triggered by some external wiring and saving them to an SD card'.

See also: almost any specialty medical electronic device ever manufactured.

[0] https://linuxgizmos.com/tiny-rugged-com-runs-linux-or-androi...


> This whole camera design looks like one of those 'we gave this project to some undergrad engineering students who've never designed a commercial product before and had no price target and thus it has a whole damn embedded linux system inside it for merely taking some HD video and stills triggered by some external wiring and saving them to an SD card'.

> See also: almost any specialty medical electronic device ever manufactured.

These are not design mistakes.

When building products in short runs and where the costs of part have little impact on your margins compared to R&D, it completely makes sense to go for a full computer rather than bother with embedded development where everything is more complicated. Medical also has to deal with certification which is a much more significant concern than saving on parts and will often reuse already certified components.


I'll admit I only watched a video on it not the report, but it had pictures reportedly redacted at manufacturer request. It showed a teensy 3 and some adafruit qwiic board in there. Obviously the real engineering is in the enclosure. Otherwise it could just be a webcam. But still, it's clearly not a very in depth electrical design. I'm all for SoMs if you can but they don't guarantee you the adventure of custom hardware bringing moving through all the software stacks and whatnot.


No serious commercial product should be using a Teensy under basically any circumstance.


Can I ask why? I'm not really into microprocessors.


Usually (not always), something like a Teensy or a Pi Pico or an Arduino is treated like a development board for prototyping.

A person builds out their circuit using hardware they can solder/wire-up by hand on a workbench, maybe even with relatively-giant solderless breadboards, to prove the concept and the general design.

And a dev board can be great for spinning a few prototypes. It's quick to get started (code can begin being tested on-chip after just plugging in a USB cable), and to try different things and to make (and correct!) mistakes. (Blow up a Teensy? No worries; just grab another from the drawer, try not to make that same mistake again, and keep moving -- no esoteric soldering required)

But when the design is finished-enough and it becomes time to spin up custom-built PCBs for a final product that will be sold, a separate dev board like a Teensy tends to lose much of its initial charm.

Instead, it's more-typical just put the microcontroller IC plus whatever supporting hardware is necessary for the overall device's actual functions on the main board. Don't need USB, or an Ethernet PHY, an LED, a button, or a separate voltage regulator? Want more or less flash? When including the MCU on a board of one's own design instead of a kitchen-sink dev board, one is empowered to use exactly the parts that are required.

This can save a substantial amount of space and greatly improve the flexibility of the layout, while also improving mechanical and electrical robustness by having fewer connections between the MCU and the world around it. Plus, fewer parts tend to be less costly than more parts are.

(But again, it's not always done this way. This camera from the submarine is an example of one instance where the whole dev board was put inside of a finished product. Sometimes that's a good idea, and sometimes it isn't. I'm not attempting to suggest that it was or was not a good move in this instance.)


Everything you said makes sense except you haven't explained why you can't just seal up a Teensy in an enclosure and sell it that way, except for "you're not supposed to do that". Are Teensies prone to random failure or something? Because if they just work and you're only selling <50 devices for extremely specialized nieche then I really don't see a problem with this?


(I previously wrote something rather crude. I am attempting to revise that into something more positive.)

In this context, all people are free to do whatever they want. It is beyond me to suggest that any person cannot do a thing.


Well, but it's not about whether someone can or cannot do something. Since you seem experienced with these devices I'm just asking if there's any technical reason why this might be a bad idea other than the fact that it just doesn't seem like a very professional thing to do. Like for example even though I'm far from an expert I know Raspberry Pis would be awful for any commercial application because they are notorious for killing their SD cards rendering the device useless.


The Teensy wasn’t engineered, tested, rated, or certified for any sort of continuous duty, let alone within a pressurized O2-enriched environment (assuming it was inside the vessel), especially not within deep sea Helium-enriched environments (that have been shown to break things like MEMS devices), and present unnecessary risks for an entirely inefficient choice (see: comment above, Teensy’s are ~$30-40/ea where small PCBs populated with the same circuit features can be had for under $20).

I’m probably not as qualified as the person you replied to, but that’s my intuition as someone with a passing familiarity with electronics engineering (I have an associates degree in EE).


People do whatever they want. It doesn't have to make sense.

Perhaps disturbingly: I even know of one bit of critical public safety communications infrastructure that is is expensive, low production volume, and has a Raspberry Pi 3 embedded inside. I won't name names because that's getting a little too close to home for my liking, but I was quite surprised to find this inside of a very nice waterproof box with chonky, expensive, olive-colored milspec connectors to connect it up to the outside world.

Which, well: Yeah. There's a ton of good reasons not to do that. But building a whole Linux system on a custom board using individual parts is hard, so it can make sense to buy someone else's work instead.

Except... that's what the CM3 is designed to provide, including on-board eMMC instead of an SD card. I'd not have been surprised at all if there was a CM3 in there, but there is instead an entire Pi 3.

But MCUs, like on the Teensy, aren't like that. They aren't hard to integrate on a custom board like the Broadcom SoC on a Pi 3 or CM3 is.

The primary purpose of an MCU is not to be stuffed onto a dev board like a Teensy, but instead to be stuffed onto the board inside of a microwave oven or an air fryer or a fancy remote control and be easy to interface with other things and to program.

It really doesn't take much to get them going: Some require external ROM or flash, but a lot of them have internal flash memory and only need power and programming pins wired up to let them run code and do whatever IO is needed within a system.

This camera already had at least one very custom board inside. It could have integrated the MCU, as well, instead of the kitchen-sink Teensy.

Doing so is not just style points; it's quite often easier, cheaper, and more flexible.

This allows a person to use all of the IO pins on the MCU to do stuff with, instead of just the functions that the designer of a dev kit decided to build out through whatever interfaces they decided to include.


That’s entirely true for manufacturing at scale. I really doubt they are selling a lot of these.


I suspect you're right about the quantities. In support of that notion, when looking closer at the (linked in another user's comment) PDF of the report, I can see that a lot of this camera's internal structure quite clearly appears to be the product of an FDM 3D printer. This suggests that quantities are low.

And I don't know when that camera was manufactured or designed.

But these days, it's possible to get even hobbyist-quantities of custom PCBs delivered with difficult-to-solder ICs installed from sources like JLCPCB.

(Depending on the features and functions wanted, it doesn't take a whole lot of extra parts to get an MCU to do its thing: There's not a ton of parts on a Teensy to begin with.)


JLCPCB will stick an IMXRT1064 and an oscillator on 5 custom PCBs for you for <$100 - the Teensy is basically $25-$35 for the same thing.


$25-35/ea, so well over $100 for 5 boards.


The 3D-printed (and hot glued?) part in Figure 3 further support this theory (not that 3D prints can't be used in production).


Indeed this is massively overcomplicated, as one only needs to see what dashcams use to know that you don't need, or perhaps want, an entire OS on it.


Does not leave SubC in a particularly flattering light...


They had no idea how their own product worked. They didn’t even know it used encrypted storage.

This was either outsourced or done by some junior engineer who was putting pieces together like it was another Raspberry Pi project that just needed to kind of work.


The longer I last in this world the more products I realize are the result of telling a few people who don't know what they are doing to "make it kind of work."


That’s my entire experience in embedded. Everything I get from other companies basically looks like an internship project right down to the pointer arguments with unspecified bounds on the function calls. One of the companies we bought hardware from keeps representing things are working when they only work on devices in the lab. Almost nobody in the space produces anything professional and everything uses Yocto even for two person projects where Multistrap would be more productive.


> Almost nobody in the space produces anything professional and everything uses Yocto even for two person projects where Multistrap would be more productive.

While I agree with your sentiment that there's a lot of poor software engineering in embedded space (especially in consumer-oriented novelty products, less so in established fields like industrial or telco), I can't but wonder what's wrong with Yocto? In my experience, it's quite the opposite: Yocto is the quickest path to get the firmware for a new device assembled, once you have climbed its pretty steep learning curve. I have built a few homebrew firmware build systems out of Debian and make/shell scripts (not my choice), you pretty quickly find yourself reinventing half of the stuff that Yocto does out of the box, but it's all bespoke, janky and hard to maintain. While with Yocto you just take the vendor's meta layer for BSP, put your application in another, and it bakes you a set of flashable images on the other end, complete with SDKs and other goodies for your dev workflow, reproducibly. It doesn't get significantly more sophisticated once you start to need kernel customizations, firmware updates with A/B partition layout, readonly rootfs, manage board- or customer-specific variants and other features that are very common in embedded systems but poorly or not at all supported in standard distros.


The problem isn’t OE per se but every vendor pulls in their own janky patches for everything and you find yourself fighting the build system to cut packages and reduce the filesystem size. It’s also really slow and it’s frankly massive, requiring tens or hundreds of gigabytes of stuff to yield less than a gigabyte of system image. And at the end of the day if a vendor breaks something from another vendor you will find yourself struggling to figure out why. Older versions of their documentation make statements that small teams should avoid Yocto for these reasons.

If you don’t already know anything about how to do these things in Linux Yocto is good for it. But if you try to swim against the current of the way the layer authors want you to do any of those things you will find it very challenging.

Take read-only root, for example. Usually it’s a one-line change in a config file plus an additional mount should you want an overlay in volatile memory. I don’t see how pulling in layers and config files does anything other than obscure how that works.

For a good example of what I hate about it, try building from Meta-Intel without graphics drivers.


What kind of hardware projects do you work on?

I'm mostly in the software space but in the past few years I've been doing a lot more embedded stuff, and the trend I notice is that companies are making great hardware, and then completely ruining its usefulness with bad software and firmware. It's kind of mind blowing to me because I always considered software to be the easy part of making a product, compared to, you know, etching microscopic patterns onto sand to make magical transistors appear in just the right way to do the task you want.


It's all about what works enough according to a rather low bar.


It's almost as if hardware was developed by actual engineers, unlike, you know, the other half of the pie.


This is in fact the case.


It survived the pressure, does the rest matter?


In this context? Certainly. This is more examples of their using untested, uncertified equipment. Anyone who’s ever worked in O2-enriched pressurized environments should at least be instantly concerned about whether this device had been properly rated for fire prevention.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: