Sign in Subscribe
By Steve Brownlee —

The Loop You Can't See

Competence comes with doubt; the erosion of competence comes with confidence.

The Loop You Can't See

The standard pitch for AI in the enterprise is that it lifts everyone. Give a team better tools and the whole team gets more done. It's a reassuring story, and it's wrong in an important way. Large language models don't raise everyone uniformly. Using them initiates a reinforcement loop. Reinforcement loops can be negative or positive—and they compound. Each reinforcement in either direction amplifies the effect even more.

What determines how the tool reinforces you? Your decision to keep ownership of the cognitive work. The moment you start believing that the Large Language Model (LLM) has the ability to perform cognitively challenging work that requires judgement or taste, you are in a negative reinforcement loop.

The same model runs upward for one person and downward for another, and the only thing that sets the direction is whether you keep doing the thinking or quietly let it go.

When you keep doing the work, the loop runs upward. You treat the model's output as a draft or a hypothesis rather than an answer. You interrogate it, catch the errors, push back, recombine.

You cover more ground in less time and spend the freed capacity on synthesis and judgment. Each exchange teaches you more about the domain and more about how to drive the tool—two skills that reinforce each other.

When you hand the work off, the loop runs the other way. The model becomes a way to avoid thinking rather than a way to think faster. You accept the first plausible answer. But the effortful struggle you just skipped is precisely where understanding forms. Skip it consistently and the underlying skill quietly atrophies, which weakens your ability to judge the next output, which makes you lean harder on the tool, which accelerates the decline.

Dependence deepens by degrees. This isn't only a worry on paper. In a study of 666 people, Michael Gerlich found that heavier reliance on AI tools tracked with measurably weaker critical thinking, with cognitive offloading as the mechanism sitting in between. A separate MIT Media Lab team wired participants to EEG sensors while they wrote essays and found that the group leaning on a chatbot showed the weakest neural connectivity of anyone in the study—a pattern the researchers labeled cognitive debt.

The dangerous loop is the one that feels like competence.

Here is the part that should worry professionals most: the downward loop is nearly invisible from the inside, and it's invisible in an asymmetric way.

The person sliding into dependence produces work that looks polished. The prose is clean, the analysis sounds authoritative, the deck is well structured. There is no error signal—nothing that feels like failure—so there's no prompt to correct course. They feel more capable, not less. The person in the upward loop, by contrast, keeps the metacognition to know what they don't know.

Competence comes with doubt; the erosion of competence comes with confidence. A Microsoft Research and Carnegie Mellon survey of 319 knowledge workers found precisely this split: the more people trusted the AI's output, the less critical thinking they did, while those who trusted their own expertise scrutinized that output harder.

This is why LLMs are more hazardous than the cognitive tools that came before them. The calculator panic, and further back Plato's warning that writing would rot our memories, were about offloading narrow mechanical functions. A calculator computes; it doesn't reason. An LLM simulates the processes of analysis, argument, and judgment—the layer we think of as thinking itself—and it produces output with a fluency that reads as understanding.

At the level of an organization or a market, the loops don't just persist—they spread apart. The people and teams already disposed to do the work pull further ahead; the ones disposed to offload fall further behind, and the gap widens faster than it would have without the tool. The technology sold as the great equalizer turns out to be a cold and ruthless mechanism that accelerates the chasm between those developing good habits and those who aren't.

This has nothing to do with intelligence. The trait that decides the loop's direction isn't raw intelligence — it's metacognition and epistemic vigilance. Those willing to put in the real work to obtain the domain knowledge needed to evaluate what the models hand you will be the ones who maximize their value.

The hardest part is that the tool will never tell you which loop you're in. It returns the same confident, well-formed text either way. The signal has to come from you—from a discipline of interrogation the technology is built to make optional. Whether AI makes you sharper, or slowly hollows you out, is not a property of the model. It's a property of how you use it, and by the time the answer is obvious, the loop has done its damage.

Sources

Previous

You are the AI agent's harness
Share

You might also like...