What is Reasoning Anyway?

Discussion about Large Language Models often involves discussion about reasoning. I have a bit of a problem with this, not necessarily that LLMs can’t reason, but that calling something reasoning tells us very little about what we can actually verify.

AI is a broad church. LLMs are its newest and by some distance its loudest congregation, but the building also holds constraint solvers, planners, theorem provers and the logic programming tradition I grew up in.

Ok, that’s a bit of a tortured analogy, but you get the point. John McCarthy, who coined the term artificial intelligence, is often credited as once complaining that “As soon as it works, no one calls it AI any more“. Technology in the field of AI has been in production doing useful work for decades. When our whole field gets described by its most recent arrival, the older members don’t stop existing.

I’ve been involved in AI for almost three decades and have had to make my peace with the word “AI” being vague. But I’m still finding the overloading of the word “reasoning” troublesome.

Around the time OpenAI released o1 in September 2024 “reasoning model” came to describe a model that produces a run of tokens before it commits to an answer. Since then the word has spread to almost everything. Agents reason. Retrieval pipelines reason. As I wrote recently when experimenting with Jev, a well built classifier now gets described in the language of System 1 thinking. Somewhere along the way the word stopped describing what a system does and started simply meaning it’s somehow clever.

I learned the meaning of the word reasoning as something much more specific than “clever system”. I wrote my first Prolog program at Sussex University in 1998, I spent a semester fighting the cut, and have spend the last twenty odd years building systems based on logic. In that tradition reasoning has a definition, you have premises, you have rules, and you have a procedure that derives conclusions from them.

All the way back in 1979 Bob Kowalski talked about Algorithm = Logic + Control. Logic is what the system knows and control is how it goes looking for an answer. The reason this is still important today is that a derivation can be checked by someone other than whatever produced it. If the conclusion does not follow from the premises, the check fails, and you can point at the step where it went wrong.¹

The chain of thought a language model produces looks like one of those derivations, which is a little troubling. It is text in the shape of an argument, generated by the same process that generates the answer. Whether it is the reasoning, or a plausible account written alongside it, is an open question. CoT tokens may genuinely contribute to finding the answer, but that doesn’t make them a faithful or independently verifiable account of how it was reached. Anthropic published a paper² in 2025 showing that reasoning models often do not say what they think. Subbarao Kambhampati and colleagues argued that treating intermediate tokens as human-like reasoning or thinking traces is fundamentally misleading.³

I do not think that makes the models useless, but it does mean a word we can’t really pin down should not be carrying weight when we specify what a system is for. If there’s one thing we can agree we’ve learned from LLMs it’s that language is important. Given that, I wish we could be more precise about what we mean when we talk about reasoning.

If AI is a broad church, and we agree language is important, then we need words that tell its members apart. Reasoning is a term with a history in our field and I would like to keep it. I’m not suggesting that statistical models can’t perform reasoning in a broader sense, but when we rely on a system to make consequential decisions I think the word should carry a stronger guarantee. In this context reasoning should mean premises, rules and a derivation that can be inspected and checked independently of the system that produced it.

That is the computational logic sense of the word, and it is broader than people tend to assume. It covers inference under uncertainty and abduction, both of which have well worked out treatments in the logic programming literature. A model that assigns 0.94 to “approve” is reporting a learned assessment, not showing which policy conditions were satisfied. That can be enormously useful, but it is quite different from applying an explicit policy and showing which conditions were satisfied.

Of-course that doesn’t guarantee the premises were right, or the rules were any good, but it does establish whether the conclusion follows from them. ↩
https://www.anthropic.com/research/reasoning-models-dont-say-think ↩
https://arxiv.org/abs/2504.09762 ↩

Footnotes