The Threat Surface

Four essays in, my position is this; the takeout model plus operator legible policy authoring solves the trust paradox by replacing vendor promises with structural guarantees.

That is a strong claim, hence the right question to ask is where does it break?

This essay walks the threat surface honestly, the attacks it handles well, the ones it handles partially, the ones it does not handle at all. The point is not to defeat the architecture, the point is to know it. If you know where an architecture breaks you can deploy it, if all you have seen is the demo you will regret it.

I will go by entry point, because that is how an attacker thinks.

Start with the diner, the agent itself is the threat. The most common attack on agent systems is prompt injection, malicious instructions fed to the agent through something it reads, a document, a webpage, a tool output, and now the agent wants to do something the user never asked for.

The takeout architecture handles this well. A fully injected agent still has no keys, no credentials, no kitchen access, it can only place orders from the menu, the cashier checks every one, the chef executes only the ticket. The blast radius is whatever was on the menu for this diner today.

It gets harder when a legitimate menu item is dangerous in the hands of a confused agent. Draft and send wire transfers I am authorized to send is a legitimate capability, and an injected agent can weaponize it. So high stakes items get friction, human confirmation, hardware key signing, out of band verification, and you design the menu as if every agent will eventually be injected, because we know they will.

There is a related threat that is mostly unsolved, compositional leakage. Two agents each scoped narrowly combine outputs into something neither was supposed to produce, agent A summarizes documents safely, agent B summarizes emails safely, agent C reads both summaries and infers something neither one meant to reveal. The receipt layer catches some of this after the fact but in the moment not really, it is an open research problem and I would rather say that plainly.

Then the principal, the human is the threat, willingly or unwillingly. This is where the agent has to be more than a constrained tool, it has to be a witness, present when the intent forms, with enough context to notice when something is off.

Take social engineering. Kenji gets a call from someone claiming to be the CFO, urgent, wire $200,000 to this new vendor, I am in meetings. Kenji asks his agent to draft the transfer, and Kenji is really Kenji, at his real desk, really authorizing it.

The agent watching the thread should notice things. The recipient was added to contacts four minutes ago, there is no history of transfers to this vendor, the CFO never routed a transfer through Kenji at this hour, and the call landed 90 seconds before the draft request. That pattern matches social engineering, and the agent should stop and ask.

The architecture supports it, the cashier can force out of band verification on weird transfers, the agent can challenge the principal, the receipt captures the whole context. But it only works if the agent was designed as a participant in the security loop instead of just a target of it, and most current agent systems were not.

The harder version is duress, Kenji authorizes because someone is standing behind him. Duress codes help, a phrase or action that means I am being coerced do not proceed, but only if they were set up in advance and Kenji can produce one under pressure. Duress is something the architecture mitigates, it does not solve it.

The hardest version is the insider. Kenji himself is the bad actor, he is leaving for a competitor and pulling out what he can before he goes, slowly, inside his authorized scope. The architecture catches this through patterns, unusual export volumes, off hours activity, access drifting outside his usual range, but not in the moment, the inspector might surface it after a week and by then the damage is done. Slow insider abuse inside authorized scope is frankly the hardest threat class for any architecture. The real defenses are organizational, two party authorization on sensitive bulk operations, behavioral monitoring with carefully tuned thresholds, and accepting some false positives.

Then the cashier, the governance layer itself is the threat. There are two flavors here, the technically compromised cashier which means a bugged build or a malicious update or a zero day in the gateway, and the misconfigured cashier which means Tanaka wrote a policy broader than she understood.

The technical compromise you defend with hardware attestation, the cashier’s code runs in a TEE, attestation proves the code matches a known good measurement, external audit checks periodically, and depth at the sandbox layer means even a compromised cashier does not get arbitrary kitchen access.

Misconfiguration is harder and way more common, most real world identity breaches are misconfiguration, not exploits. The defense is an authoring interface where mistakes are hard to make, sensible defaults, warnings on broad grants, curated templates for the sensitive stuff, two admin approval on high stakes changes, simulation before deploy, which is the Tanaka problem again. Whether the takeout architecture is safe in practice mostly comes down to whether her console makes the right choices easy and the wrong ones hard, which in my humble opinion is a product design problem that everyone keeps treating as a security problem.

Worst case is the complicit admin, Tanaka herself is the bad actor or shares credentials with one. Separation of duties helps, the admin who authors policy is not the admin who reviews the logs, plus two admin approval on the most sensitive changes, and if multiple admins collude the separation breaks. This is mitigated again but not solved, the full defense is organizational.

Then the kitchen, the data itself is the threat. The architecture assumes the kitchen is honest, the documents are what they claim, the records were not tampered with, the upstream feeds were not poisoned, and that assumption is wrong in general. Data at rest gets modified by attackers who got in some other way, feeds get compromised. So the kitchen has to be cryptographically honest, every record hash signed with verifiable provenance, every external feed with chain of custody, the chef verifying integrity before serving. Most enterprise data stores do not have these properties today and retrofitting them is expensive, so deploying the takeout model honestly means spending on the data layer, and the agent infrastructure conversation mostly glosses over that part.

There are covert channels too, even with perfect isolation an attacker who controls part of the kitchen can leak through timing, resource consumption, error messages. Side channel resistant primitives reduce it, they do not eliminate it. The honest position is the architecture raises the cost of exfiltration, it does not zero it.

Then the chef, the sandbox is the threat. The chef is a Firecracker microVM or a gVisor sandbox or an attested enclave, and the architecture assumes it stays in the kitchen. Sandbox escapes exist, they get found, patches ship. The defense is depth, multiple isolation layers, limited host capabilities even after an escape, attestation that notices when a sandbox is not what it claims, and the price is performance and complexity.

There is a subtler problem too, sometimes the legitimate output is itself sensitive. A menu item that says summarize this confidential contract returns a summary full of confidential information, and output filtering cannot help because the output is supposed to be sensitive. That defense lives in menu design, sensitive items get strict authorization, scoped principals, extra friction.

Then the receipts, the audit layer is the threat. If the logs can be tampered with, the whole accountability story collapses. Hash chaining defends that, editing an old receipt breaks the chain visibly, and external anchoring, committing the chain root periodically to a public ledger or a third party witness, blocks private manipulation by anyone including the platform operator.

The harder problem is overwhelm, the system records everything so it records too much, and the real signals drown. Detection has to be tuned, false positives burn operator attention and false negatives let attacks through.

And the deepest problem, all audit is retrospective. Perfect detection still finds the attack after the damage. This architecture limits blast radius and speeds up detection, it does not eliminate breach.

Last is the building itself, the meta threats. Some threats go after the architecture’s existence instead of its operation. The kitchen burns down, that is regular disaster recovery. A government subpoenas the keys, a regulator demands access, assets get seized in a hostile jurisdiction, that is a political problem, the architecture can constrain it but cannot eliminate it. And for anything meant to run in regulated industries the sovereignty question matters way more than people admit. Where does the kitchen physically sit? Who can compel the keys? What happens to the receipts if the platform vendor changes its mind? These are policy questions, not engineering problems, and they need answers before deployment.