C2PA Cameras Do Not Survive Contact with Reality

I have ordered a faulty Sony A7 IV motherboard, but due to its faulty-ness and the lack of the rest of the camera, I'm not sure how far I'll be able to get with it.

And, Apple could choose to integrate a LiDAR depth map into the signed photo as a mitigation against the analog attacks (eg. pictures of screens).

The provenance “proof” these approaches provide is very tenuous and nowhere near the “this is a real photo of a real world event taken by a real camera and not an AI image” proof that marketing types like to push.

Apple doesn’t want a PR disaster where some crazy image is totally fake but becomes world news because it is “cryptographically signed as being from a real iPhone so it must be real!”

>Images captured with an opt-in Reference mode can be authenticated to confirm they were taken with an iPhone. Authenticating is done by tapping the Reference badge on the image, which sends the raw image, sensor signatures, capture time frame, and the unique hardware identifiers of the sensor to Apple's Private Cloud Compute (PCC) servers. PCC uses the information to determine whether the camera captured the photo, gives it a unique ID, and then returns an authenticated version to the user's device.

If it does ship like that, it’s hard to not imagine a situation as I described earlier - an “iPhone Reference Image” being used to propagate fake news, at which point the credibility of the feature goes to 0 (and Apple’s takes a severe hit).

Wait & see.

But:

- You cannot prove that no one has ever been able to break a Secure Enclave or a YubiKey or another secure element. That's okay, these companies focus on making it so expensive that it's not worth doing.

- The analog "attack vector" is real, but that was always true with analog cameras as well. Anyone could have taken an analog picture of a screen, or even painted a hyper realistic image of something that never happened.

I think a realistic goal for provenance is to at least get to parity with analog cameras ("this is a picture directly from a sensor"), not to make the perfect system for proving that a photo is of a real world event.

At over a decade old, still prescient as ever: https://www.youtube.com/watch?v=HUEvRyemKSg

RMS must be having daily nightmares at this point.

P.S.: Fantastic talk you linked there.

Doesn't sound like it's engineering-driven, even though they still do have a lot of capable engineers sitting there and atrophying.

I also wouldn't rule out that the less capable ones actually believed that the systems they've built are unrootable or something like that.

I don't think completely solving this sort of problem is even possible.

The analog hole is alive and well:)

You will not build a perfect system, or even something near perfect. The best you're going to do is make it so that it's hard to casually present AI photos as real, leaving only the cases where it really matters. In the "best" case, you've just made the public more trusting of photos in general, so that when there's actual money or power on the line that makes jumping through the hoops to fake authenticity worth it, the public is more susceptible.

The best outcome at this point is for everyone to get on the same page that photos have roughly the same probative value now as drawings. Poorly thought out snake oil efforts to prove authenticity are only going to delay that.

This is ridiculous.

How?

Right now, the Internet training set is becoming more and more contaminated with better and better generative AI images and video.

It makes the models more screwed up, and makes it very difficult for humans to figure out what is original and not too.

If there was some signal that could at least make it easier to identify ‘original’/real images…

The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.

I remember the original paper showing issues with even a couple percent of certain kinds of synthetic data too, not 100%.

People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless.

"C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.

Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.

When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.

Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.

The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.

While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.

This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.

I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.

It is however in the interest of the people to keep the systems running in an untainted way.

As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.

C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors". Banality of evil. Again.

___

Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.

Those are different things. Argh and I ran with your term aah

Not to mention, democracy is under just as much or more threat from "banal" fake news created by citizens. People gonna people. They don't need the DPRK lying to them to fool themselves.

And I agree on state actors. If a major one is invested in something like this they might have well compromised the signing project itself, the verification process, or even the court system or media. That seems like a rare and extremely high bar to guard against.

Yes, absolutely. Probably moreso. The whole point of these proposals is to try to solve for the "motivated malicious user" who is engaging in actual high-stakes fraud. There is no point in applying techniques that suppress inconsequential pranks while making serious crimes easier to get away with.

This really seems like a rehash of the perennial DRM argument: DRM restrictions provably do not reduce large-scale motivated copyright infringement, they just annoy legitimate paying users. This is the same class of solution, in that it is effective only where the stakes are low and the impact is minimal.

Also I'd like to know if a "joke" is likely fake.

Bur I don't even really think that's really relevant anyway, because whatever the density of "malicious" motivations is, the point here is that the fact that it only is an effort/motivation threshold that allows this technique to "block" malicious uses, and the motivation to overcome that threshold correlates directly with the stakes involved in the malicious use.

In other words, the more malicious the abuse is, the less effective this solution will be: the boundary of its usefulness will be wherever the line between pranksters and actual criminals happens to lie.

Your argument applies to any imperfect security technology -- aka practically all of them.

No, my idea of a criminal is someone who is motivated to commit crime, and I feel that we've already established in this thread that the approaches we're discussing are motivation gates far more than competence gates.

> Smart, dedicated, technical people can typically make more money legally.

Then who's been running all the botnets, writing cryptolocker malware, and running phishing scams for the past couple of decades?

We've always had script kiddies, and now we have people using AI itself to do malicious things. Technical skill has never been an obstacle for sufficiently motivated scammers.

> Your argument applies to any imperfect security technology -- aka practically all of them.

Ultimately, everything has weaknesses, and with enough effort, most measures can be circumvented. But how much effort is enough varies wildly between solutions.

There's a huge gulf between a "no trespassing" sign, on the one hand, and a concrete wall topped with barbed wire, on the other. The "no trespassing" sign only keeps out people willing to obey it; the concrete wall keeps out anyone who isn't willing and able to accept the time, effort, and risk necessary to climb over it or knock it down.

And the point is that using digital signatures to distinguish AI-generated media from hand-made media is much closer to the "no trespassing" side of things than it is to the wall. Maybe it's analogous to a gate with a latch you can open from the other side if you reach over in just the right spot.

* Pangram: Yes we should really be discouraging people from putting trust in tools like this because they can't be made totally reliable.

* Antivirus: We should be building application environments with robust security models so that malicious software has a limited blast radius (like we do on mobile, like the Linux ecosystem is trying to do with Flatpak, etc).

* HTTPS: HTTPS is a strict upgrade from HTTP so we should be using it everywhere possible. The UI symbols to indicate to users the security expectations they're getting are good practice.

* ssh: This is just an inappropriate comparison.

The HTTPS comparison would make more sense if actually 0.1% of the time when their browser said they were using HTTPS it was just lying.

Antivirus: "Actually, we should build this hypothetical better thing" is a cop-out.

HTTPS: C2PA is a strict upgrade from unsigned photographs, so it should be used wherever possible.

SSH: Totally appropriate, the entire point of the discussion is whether it's permissible to the user that they might be more secure.

Also, note that C2PA should have something like Level 3 as well: "The image is mathematically proven to have come from the physical camera sensor."

It's somehow difficult to achieve this, but it's possible (although the attacks will always be possible of course).

But yeah, difficult too :)

The screen is doing all kinds of crazy things that are not apparent to your eyes, but that show up clearly on camera.

For example, you'll see a sort of barber-pole effect when pointing a video camera at a raster-scan digital display whose refresh rate isn't synced to the camera's frame rate. To avoid that, sync them, or maybe use a colour e-ink display.

Alternatively, print a high-resolution version with a decent photograph printer and take a picture of the print with the C2PA camera.

Exactly: most people don't bother editing visible watermarks out of AI-generated media, because they have little or no incentive to bother doing so. This will "defeat" the 99.9% of users who are not actually trying to do anything malicious, but will be a minor annoyance to the 0.1% of users who are actively engaging in fraud, fabrication of evidence, etc.

The upshot is that not only us this not useful for its intended purpose, it will lure people into a false sense of security by creating expectations that AI-generated media will always be easily identifiable as such, and reduce the level of scrutiny that gets applied to the stuff that actually is malicious.

Everyone realizing that photos don't prove anything would defeat 100% of malicious users. I don't understand what people incorrectly trusting photos is supposed to achieve at this point, in your view.

Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.