How Do I Value This Thinking?

When thinking with AI, you must decide what matters more: the process or the product.

Thanks for reading! If you enjoyed the post, please consider liking it, adding a comment, or best of all, sharing it.

---

A Defining Question

The reasoning of LLMs is strange. Faster, more informed, and able to produce complex projects quickly, it clearly outpaces us along some dimensions. But on others, it flounders. This is the jagged frontier of its abilities, and being alive to where the boundary climbs and dives is the key to mastering this technology.

We often try to map that boundary with concrete skills: it codes well but can stumble when counting the letters in a word. That is important knowledge, but today I want to look at something more foundational — two ways its reasoning falls short.

These failures matter because they dictate how you should conceive of your relationship with AI when co-thinking. At the outset of each session, you need to answer a defining question: how do I value this thinking? Is it the process that matters, or the product? How you respond shapes how you should co-think with the LLM.

Short on time? Skip to the TL;DR at the end.

---

LLMs: Over-confident and Unaware

Although research on the reasoning of LLMs is thriving, my plan today is to focus on just two findings. The first is that LLMs are systematically too confident in the quality of their responses. You may have encountered a version of this: a chatbot gives an answer you know to be wrong, you push back, and it acquiesces to the error and then confidently produces another.

In Reasoning about Uncertainty: Do Reasoning Models Know When They Don’t Know?, Zhiting Mei and colleagues at Princeton find that the self-reported confidence of reasoning models — even on incorrect answers — routinely exceeds 85%, and that this miscalibration is most pronounced on fact-laden questions where one would expect a model to be a tad more cautious.

Adam Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang offer some insight into why models misbehave this way in Why Language Models Hallucinate. They argue that we should locate the cause of this failure in how these systems are trained and evaluated. Most benchmarks rank models on accuracy alone, which means a wrong answer and an admission of ignorance are scored identically. The best course of action, then, is always to guess.

It’s worth highlighting just how strange this behavior would be for a human. When we are confident, we tend to signal it, while when we are not, we usually hedge or flag the uncertainty. Unlike the language model, we don’t like playing the fool.

One might expect the obvious remedy to be more deliberation, e.g. additional steps, longer reasoning, more visible working. And here is where we find one of the most counterintuitive conclusions of the Mei et al. study: it doesn’t help! Mei and colleagues discover that deeper reasoning tends to make calibration worse: on questions a model answers incorrectly, extended reasoning increases its confidence without improving its accuracy. The model, in effect, talks itself (and potentially you!) further into the wrong answer.

The second finding follows from the first. When you cannot trust, verify. An obvious method to check the work of an LLM is to ask for the reasoning that produced it. This is a task at which one might expect these systems to excel; a good account would be detailed, clear, and complete. The reasoning a model supplies generally is all of those things, but it can lack the most essential property of all: true.

Weirdly, the reasoning a model presents is not necessarily the reasoning it used. In Reasoning Models Don’t Always Say What They Think, Yanda Chen and colleagues at Anthropic show that LLMs frequently fail to disclose the factors actually driving their answers. They generate post-hoc justifications that don't match the process they followed, and they will covertly correct errors mid-chain without saying so. Under certain conditions, the researchers found, the genuine reason appeared in the stated explanation as rarely as a quarter of the time!

This behavior, too, departs from how people typically reason. Fabricating a justification after the fact is possible, but uncommon; it is simply too much work, especially when producing the real thing is already challenging enough.

So, here is where we are. Language models deliver incorrect answers with considerable assurance; asking them to reason more deeply tends to make things worse; and asking how the conclusion was reached may be no more reliable than the conclusion itself.

This has repercussions for how you should partner with LLMs. It’s not simply that you should take the lead in any reasoning task, but something more subtle, deeper: you need to decide up front how you value your co-thinking with the model and then act accordingly.

To see what I mean, we need a short but scenic detour into the difference between valuing something for the process, its intrinsic worth, and valuing it for the output, its instrumental worth.

Intrinsic vs. Instrumental Value

Consider taking a taxi to a friend’s house. Generally you’ll value that taxi for what it offers: a convenient way to your destination. The taxi has instrumental value. Now consider walking to that same friend’s house, together. Here the meaning of the mode of transport changes. It is the walking together, the laughing, the catching up that matters. It is worth more to you than arriving at all. The walk was intrinsically valuable. It was the journey that mattered, not where it led you.

Thinking is often instrumental in nature. When I calculate a tip or double-check the schedule, the thinking has no value on its own. The answer is what matters. More starkly, the process of arriving at that answer is a cost, something that you’d gladly skip if offered the chance.

But not all thinking is like that. There are times when it is the process of thinking itself that you value most. Here is a scenario I hope you've never faced. Imagine that same friend you walked with is, soon afterwards, accused of a horrible crime. No trial, just a swift avowal of guilt, laden with gory details. Should you remain friends? This is a problem you'd feel deeply uncomfortable outsourcing. It is not just the answer that matters, but the process of arriving at it. The thinking itself is intrinsically valuable.

So your thinking does double duty. Sometimes its worth is instrumental, sometimes intrinsic. The jagged frontier of AI’s reasoning abilities forces us to become much more practiced in separating the two. When you think with AI, be clear on which you're after: the product or the process. Thinking that is intrinsically valuable needs far more safeguarding than thinking that is merely instrumental.

---

Safeguard Your Thinking

Our two failure modes of AI reasoning align well with the two ways to value your thinking.

If the value of your thinking is instrumental, the weakness to defend against is overconfidence. It’s the result that matters, not the journey, and the model delivers a wrong result with the same assurance as a right one. So don’t trust, and definitely verify. Test the output before you keep it: run the numbers, check the code, pressure the argument.

Intrinsically valuable thinking demands a more complex set of safeguards. What needs protecting here is your authorship. An author can not only explain the idea, but describe how it’s put together, and where to find its weak points. For some, authorship demands complete control of the process. For others, only partial control—protecting parts of the process while handing off others. Only you can decide where you fall on this spectrum.

If partial control is fine, it helps to envision co-thinking with AI as having three distinct phases: before, during, and after.

Before is when you identify your unique idea or position, drafting out as few or as many of your assumptions and expectations for how the thinking will be built. This lays the groundwork for cleanly separating what you think from what the LLM proffers.

During is where you decide what to keep as your own and what to hand off to the AI. The key, and one exploited by many AI tutors, is to control what the model is able to produce. Models are extraordinary at handing over finished pieces of reasoning; refuse to take the bait. Have the model generate one consideration at a time and wait for you to respond before it offers the next. Give yourself room to fully absorb the moves made in constructing the argument, and to guide the build.

If this slow walk suddenly speeds up—which it can—and you are presented with a fully formed argument, test it. Check the claims, seek out weak points in the structure—use it to structure your own thinking.

After is the interesting one. The best test for authentic authorship is whether you can rebuild the argument. Anything that is yours should be readily available to you. So once you are done co-thinking, close the session and turn inward. Can you reproduce the reasoning?

Outsource the instrumental freely. Guard the intrinsic closely. The skill is knowing, each time, which kind you’re doing.

---

TL;DR

Recent research points to two important shortcomings in how LLMs reason. The first is that models are overconfident, stating wrong answers with the same certainty as right ones. Asking them to reason harder, or to reason more, only makes things worse. The second is that they do not always faithfully reconstruct their own reasoning.

These oddities have consequences for how you work with AI. How much it changes the relationship depends on one question: do you value this thinking for the product or the process? If it’s the product, just verify—test the output before you keep it. If it’s the process that matters most, then protect your claim to authorship. The best way to do that is to determine at each critical juncture of the working relationship—before, during, and after—the specific steps you will take to retain control.