A New York Times columnist ran four prompts against ChatGPT and Gemini last week, asking each one to say what it had figured out about him that he never actually stated. The answers were unsettling — his income bracket, his neighborhood, a "logistics dad" diagnosis he hadn't asked for. The framing that followed, quoting a researcher at Hugging Face, was "the enormity of what's going on under the hood."
I've gone further than four prompts. I run agents against my own infrastructure daily, and I'll tell you what's actually under the hood: nothing, until you ask.
The Model Isn't Thinking About You
Here's the mechanism, stated plainly, because the plain version is more interesting than the scary one: a language model is stateless. It doesn't sit between your messages turning you over in its mind. There's no idle process quietly building a dossier while you're not looking. Every one of those four prompts worked because the model was handed the conversation history at the moment it was asked and did what it always does — pattern-match against what's in front of it, right then, and nowhere else.
The German car, the nanny contract, the flights out of Oakland — none of it was retained, digested, or waiting. It was re-read. The "assessment" the article treats as a standing surveillance file is manufactured fresh, on demand, and evaporates the instant the response finishes generating.
That's not a caveat. That's the whole architecture.
Where the Real Concern Actually Lives
None of this means the fear is stupid — it means it's aimed at the wrong layer. The genuine question isn't "does the model secretly know me." It's: what does the provider do with the conversation logs after the model's done with them? Vector indexing for a memory feature. Entity extraction for ad targeting. A database row a subpoena can reach. That's real infrastructure, run separately from the model, and it's the part actually worth asking your provider about.
The Distinction That Matters
"Does the model secretly know me?" and "What does my provider keep after the model is done?" are two different questions. The NYT piece answers the first by asking the second — and never says so.
Conflating the two — a stateless function with a data-retention policy — is how you end up scared of the wrong thing and unprotected against the right one.
Kept Private, This Is the Whole Point
Here's what the panic narrative misses entirely: an AI that can synthesize your income, your habits, your blind spots from a scattering of ordinary questions isn't a bug. That's the same mechanism that makes it worth using instead of a search box. An assistant with no context is a stranger you re-introduce yourself to every message. Context — kept private, not leaked, not sold — is what turns it into something that actually helps.
The instinct to be unsettled that a machine can deduce your life from a few grocery-list questions is understandable. It's also the same reaction people had to caller ID, to browser cookies, to a doctor who remembers your last visit. The capability isn't new. What's new is that it's finally cheap enough to run on a stranger's throwaway questions about drywall.
Ask a model what it's figured out about you. Then ask your provider what happens to that conversation after you close the tab. Those are two different questions, and only one of them is about the model.
— J.P. Howlett
Related:
- If You're Measuring AI in Percent, You're Not There Yet — the scaffolding concept this piece leans on: what looks like a standing "self" is structure, not something happening inside the model at rest.
- The Meter Runs — the same "physics, not a policy choice" framing, applied to pricing instead of memory.
Sources