The author of this article purports to have done a ton of research into this system, but appears to have missed basic information that I’ve acquired from a few podcasts.
Namely the “1-in-a-trillion” false positives per account per year is based on the likelihood of multiple photos matching the database (Apple doesn’t say how many are required to trip their manual screening threshold).
So they are assuming that the photos are independent. Common error with probabilistic reasoning. What if the same photo (or photos of the same scene taken seconds apart) gets uploaded more than once? The likelihood no longer multiplies. I don't believe the "one in a trillion" claim.
They say that they address this issue through a mechanism outside of the cryptographic protocol, but don't say specifically how. The quote from the paper:
A user might store multiple variants or near-duplicates of the same image on their client. In our language, this means that a single client could hold two triples (y,id,ad) and (y,id′,ad) that have same hash y, but different identifiers. This causes an issue that is addressed outside of the cryptographic protocol. Suppose a user copies a single image from a USB drive onto his or her device. The image will be assigned an identifier id. Later the user copies the same image from the USB drive onto a different client device. The new copy of the image will be assigned a new identifier id′ which is likely to be different from id. Because the two copies have different identifiers they will count twice towards the tPSI-AD threshold. In particular, the two triples will cause two distinct Shamir shares to be sent to the sever, even though they correspond to the same semantic image. Several solutions to this were considered, but ultimately, this issue is addressed by a mechanism outside of the cryptographic protocol. [0]
Namely the “1-in-a-trillion” false positives per account per year is based on the likelihood of multiple photos matching the database (Apple doesn’t say how many are required to trip their manual screening threshold).