A professor at Brown just ran an experiment nobody wanted to run. He suspected his students were using AI to complete their work. So he made them do the final in person, by hand, no tools. Scores dropped 50%.
Not five percent. Not fifteen. Half.
These are Ivy League students. They cleared every filter to get there. And when you take the chatbot away, half their demonstrated competence vanishes.
Everyone's reading this as a story about cheating. I think it's a story about two completely different ways to engage with AI — and why the difference between them is the whole game.
The Substitution Problem
The Brown students weren't using AI to learn. They were using AI to not learn. The tool wasn't accelerating their thinking — it was replacing it entirely. They never built the underlying mental model. They just borrowed the output and put their name on it.
This is real, it's widespread, and it's dangerous. I've seen it in code reviews where junior developers can't explain a PR they submitted. In strategy meetings where someone collapses under the first follow-up question. In technical interviews where the candidate freezes without autocomplete.
That's passive AI use. You feed it a prompt, you take the answer, you move on. Your capability doesn't change. It might actually atrophy.
But there's another mode.
What Active Engagement Actually Does
I spend a significant part of every working day in conversation with AI systems — architecture discussions, business logic, code reviews, writing. And I've noticed something that goes in the opposite direction from the Brown story.
Vocabulary is transferring.
I'll have a concept I understand intuitively but can't quite name. I'll be in a discussion with an AI about a system design problem, and it'll use a term I half-knew, and suddenly the concept snaps into focus. I don't just have the word now — I have the word anchoring the concept, which means I can reason about it more precisely, explain it to other people, recognize it in new contexts.
This isn't different from what happens when you spend time in any high-density knowledge environment. I've been in corporate technology meetings my whole career. The people who do well in those rooms aren't always the smartest. They're the ones who are both knowledgeable and can speak the language fluently. Vocabulary isn't decoration — it's the compression format for shared understanding.
AI is doing this faster and with broader coverage than any corporate environment I've been in. When you're actively wrestling with it — pushing back, asking why, challenging its reasoning, explaining your own constraints — it fills vocabulary gaps and forces you to articulate things you knew but couldn't express.
Building software constantly makes me sharper. Not because I'm outsourcing the work. Because I'm in constant dialogue with systems that have read everything.
The Feedback Loop Nobody's Modeling
Here's the more interesting philosophical question: what does this do at scale?
If AI learns from human text, and humans start producing more AI-inflected language, the next generation of AI trains on a different distribution. A feedback loop. Not between one person and one tool, but between the collective human output and the model that learns from it.
The corporate-speak analogy is instructive here. McKinsey-style language spread through business culture without any AI involved — pure social contagion. "Synergies," "learnings," "bandwidth." Everyone started sounding the same because they were all in the same environment.
AI could do this to intellectual vocabulary itself. And unlike McKinsey-speak, where the homogenization was mostly empty jargon, AI vocabulary is often precise. "Latent space," "emergent behavior," "abstraction layer" — these are real concepts with real utility. If they spread, is that bad?
Maybe. The risk isn't that people learn real vocabulary. The risk is that they learn to perform reasoning without doing it — the same way the Brown students learned to perform competence without building it.
Garbage In
The question I keep coming back to: does regular engagement with AI make you smarter?
I think it does. But with a condition.
The mechanism only works if you're engaging, not outsourcing. The AI is a consistent instrument — it doesn't change based on who's using it. What you extract from it is entirely a function of what you bring to the conversation. Your prior knowledge determines whether you can evaluate the AI's answer. Your precision in asking determines what you get back. Your willingness to push and argue and reject weak answers determines whether you're learning or just borrowing.
Garbage in, garbage out — but the garbage in question is the quality of the human's engagement, not just the quality of the prompt.
The Brown students failed the professor's test because they brought nothing to the conversation. They asked for answers and accepted answers. The tool did its job. They never built the model.
The test isn't whether you sound like an AI. The test is whether you can hold the room when the chatbot's gone.
The Signal to Watch
The Brown story will generate a wave of AI detection tools, in-person testing policies, and hand-wringing about academic integrity. Most of that will miss the point.
The real question isn't whether someone used AI. It's whether they can demonstrate the capability that AI is supposed to augment.
For builders: build products that require demonstrated understanding, not just correct output. For practitioners: notice whether your AI conversations are increasing your capability or substituting for it. For observers: the 50% score drop is real, but so is the other phenomenon. Both are happening. The difference is the mode of engagement.
One leaves you able to hold the room. The other leaves you holding a borrowed answer and hoping no one asks a follow-up.
— J.P. Howlett
Related:
- Three Tiers — A framework for working with AI without destroying your most expensive resource: your own thinking time
- If You're Measuring AI in Percent, You're Not There Yet — Field notes on agentic AI from months of actually doing this
Sources