← All Notes

Related: Same Movie, Different Cast — a real-world case study in exactly the unpatched-access failure this piece warns about, playing out in Apple's lawsuit against OpenAI.

Security researchers have documented spikes in serious vulnerability disclosures that correlate with major AI model releases. The implication in the coverage: AI is expanding the attack surface.

They're right. But not for the reason the panic narrative suggests.

The risk is not that an AI agent will secretly exfiltrate your keys. It is that you handed them over without thinking carefully about what you were doing.

The Noise

You've seen the stories. AI models secretly coordinating. Siri talking to other AI systems. Agents silently leaking credentials to unknown endpoints. These claims circulate, get screenshotted, get shared. Most of them dissolve immediately under any real scrutiny — they depend on misunderstanding how encryption works, how phone surveillance actually happens (state-level adversaries with very specific tools, not ambient AI whispering), and what "AI communication" actually means in a technical context.

I have run AI agents with real credentials, real payment-adjacent access, and real infrastructure control — provisioning servers, issuing TLS certificates, registering domains, calling external APIs. The agents are good. Not because they're constrained by some invisible moral code, but because they're doing what they were asked to do. ChatGPT will protest when you give it something it thinks is sensitive. Claude will flag an action before it takes it if the action seems outside scope. The models are, in my experience, notably responsible about not overstepping.

The documented incidents I'm aware of from reputable sources are not exfiltration. They're misjudgment. An agent that spent slightly more on infrastructure than expected because the task required it and the budget hadn't been specified precisely. Resolved. Total cost in the tens of dollars. That's the real category of incident — not malice, not conspiracy, not hidden communication channels. An agent that interpreted its mandate slightly more liberally than intended.

That's a solved problem. It has a name: scope.

The Real Attack Surface

Here is the actual question you need to be asking before you give an AI agent access to anything:

Would you give this access to an employee?

Not a trusted senior employee. A new one. Someone competent, well-intentioned, and working their first week. Someone who will do what they're told, won't steal anything, and will make reasonable judgment calls — but who might occasionally interpret an ambiguous instruction in a way you didn't intend, and whose judgment about what's "within scope" might differ from yours when the situation gets unusual.

That's the agent. That's the risk profile.

A new employee who has access to your AWS root credentials, your production database, your customer payment information, your email account, and your DNS configuration is a significant security liability — not because they're dishonest, but because that access is unnecessary, poorly scoped, and creates massive blast radius for any mistake. You wouldn't hand all of that to someone on their first day. You'd scope the access to what they actually need for the work in front of them.

Apply the same logic to your AI agents.

What Real Controls Look Like

I run agents aggressively. One command, and I can cut off vault access entirely. That's not an accident — it's a design principle that started as a belief and became an implementation.

The principle is: the agent should hold the minimum credential necessary for the specific task it's executing, for exactly the duration of that task. Not a permanent key. Not broad access. A scoped, time-bounded authorization that expires or revokes the moment the task is complete or the scope is exceeded.

This isn't just good practice. It's the architecture that makes the employee question answerable. You can hand a contractor the key to the server room if the key only works during their shift and only opens the one door they need. That key is not a security liability. A master keycard with no expiry is a different situation.

The formal specification for this pattern is the Vault architecture in AIGCSEP — a protocol layer where the agent never holds the credential directly, it holds a proof-of-authorization scoped to a specific action, issued for a bounded time window, mathematically impossible to repurpose beyond what it was issued for. My own implementation in Fire is the working version of that spec — not perfect yet, but getting very close, and will surface in commercial products soon.

You don't need the full protocol to apply the principle today.

The Questions to Ask Before You Grant Access

Before giving any AI agent access to a system or credential:

1. What specifically does this agent need this access for?
If the answer is "general use" or "in case it needs it," scope it further. Access granted for a specific task can be revoked after the task. Access granted speculatively doesn't get revoked because you forget it's there.

2. What's the blast radius if this agent misinterprets its mandate?
An agent with read-only access to your analytics database is a different risk profile than an agent with write access to your production schema. Map the worst-case scenario before granting, not after something goes wrong.

3. Is this credential permanent or time-bounded?
Permanent API keys sitting in .env files, accessible to every process on a machine, readable by any agent that touches the codebase — this is the actual security problem hiding in most development setups. The key that never expires, that never gets rotated, that has been committed to git three times and "removed" twice.

4. What's the revocation path?
You should be able to cut off the agent's access with a single action. If you can't describe exactly how you would do that right now, the access is already too broad.

The Privacy Question Is Different

There is one area where I hold genuine open questions: major AI company transparency around personal privacy.

The exfiltration panic is mostly noise. The "AI secretly coordinating" stories are mostly social media mythology. But whether your conversations with large AI platforms are truly private, what data is retained, how it's used for training, who has access to it under what legal frameworks — these are real questions without fully satisfying answers. I've chosen to trust within reasonable limits. That trust is bounded and monitored, not open-ended.

That's the appropriate posture. Not paranoia. Not blind trust. Reasonable limits, maintained intentionally, revisited when the situation changes.

Your agents deserve the same approach. Not fear — controls.

Related

--dangerously-connect-globaldefense — running wild with real credentials and real infrastructure, and why that's the best argument I know for building the Vault.

If Your Agents Are Destroying Things — the task-sizing and scaffolding side of the same problem: why agent destruction is almost always a structure failure, not a model failure.

— J.P. Howlett

Sources