The Trust Paradox

I wanted to build a Chrome extension.

The idea came from something I see every single week, someone in my feed complaining about a service they signed up for. The trial that auto renewed, the cancellation that needs a phone call to a number that only takes calls on Tuesdays, the privacy policy that turned out to be a license to sell their browsing data to anyone with a credit card. And the arbitration clause that means they can never join a class action even if the company defrauded a million people the exact same way.

Nobody reads the terms of service. Honestly nobody can, they are written in a way that reading them feels like work, so people click I agree and find out months later what they agreed to.

So the fix seemed obvious to me. A Chrome extension. You land on a signup page and it reads the terms in the background, runs them through an AI and shows you the red flags before you click. Heads up, this trial auto renews at $29/month, cancellation needs a phone call, they sell your data, the arbitration clause waives your right to sue. That’s it, five seconds of warning right before you click.

I started sketching it out, manifest v3, a content script, AI on the backend, a cache for the popular services. A weekend project, build it, put it out for free and move on.

Then I got stuck.

For the extension to work it has to be on the network, it reads the page you’re on, sends some of it to an analysis service and reports back. So to protect you from internet companies it has to become an internet company itself. To warn you about services that ship your data to third parties, it has to ship your data to a third party.

The defender has the same architecture as the threat.

And what exactly am I asking a user to trust? Just my word, my privacy policy, my promise that I am not logging what they sign up for, that I am not selling the fact that this person is shopping for life insurance to some advertiser, that I am not training a model on their contracts. My word, in an environment where every promise like that has been broken a hundred times already.

Why would anyone trust that? I wouldn’t trust it myself.

And the more I thought about it, it is not just my extension, the same problem is everywhere once you start looking. The password manager that protects you from phishing has to read every page you visit to know when to autofill. The AI assistant that drafts an email to your lawyer has to read the email. The browser agent that books your travel logs into your accounts and sees your saved cards, and you are trusting it not to do anything you didn’t ask for. The tax software that finds you deductions is reading your bank statements, and the only guarantee it won’t share them is a promise.

Every one of these tools needs access to exactly the data it is supposed to protect. The watcher needs a watcher, and the only thing standing between the user and disaster is the company’s word.

And the industry decided that this word is enough, which if you ask me it is not. It survives because nobody has been catastrophically wrong yet, and because the alternative, building software that doesn’t need this trust in the first place, is genuinely hard.

That right there is the trust paradox. To be useful the tool has to see what we wanted protected, and to be safe we have to trust the tool with the one thing we don’t trust the rest of the internet with.

Now this paradox is older than AI, your bank always saw your money and your doctor always saw your body. But two things changed.

The first is scale. AI tools want to see everything, your email, your contracts, your conversations, your medical records, your kid’s school portal. No category of software ever asked for this much before, not even close.

The second is autonomy. Old software you picked up and put down, you opened the bank website, checked the balance, closed it. AI tools now sit inside your workflow and act in the background, so it stops being I open the tool when I need it, and becomes the tool runs alongside me and I trust it.

Put scale and autonomy together and it is a different thing entirely, and we simply do not have the infrastructure to hand out that much trust to that many tools on company promises.

So either the agent era gives us the worst run of privacy and security disasters in software history, which honestly might already be underway, or something changes in how the software itself is built. And I mean the actual construction, a SOC2 report does not fix this and neither does a policy document.

The extension I wanted cannot be built responsibly in the current model, not by me and frankly not by anyone. Same goes for a hundred other tools millions of people use every day without thinking about what they are trusting.

That is essentially what this series is about.

I don’t have a complete answer, nobody does yet, but serious researchers are converging on the shape of one. What I have is a way of thinking about this problem that I haven’t seen written down in one place. Some of it comes from the technical literature, and a lot of it comes from twelve years inside regulated industries, watching them try to deploy software they didn’t fully trust. I have seen that struggle up close.

In the next essay I will make the structural argument. Agents today are running without something every previous era of computing eventually had to build, an operating system, not the consumer kind but the layer that decides what software is allowed to do, for whom, and with what evidence.