CoT’s Trust Boundary: Exploring Motivated Reasoning in Claude 3.5 Haiku

by
Table of contents

🌏 閱讀中文版本

When CoT Stops Explaining: What Anthropic’s Research Shows

The eval pipeline finished.

You open the log and inspect the CoT. The logic is tight. Every step looks reasonable. But when the model gives the wrong conclusion, the CoT still looks “completely correct.”

The key is not just the prompt. It’s a mechanism.

Anthropic published attribution-graph research on Claude 3.5 Haiku. Using Attribution Graphs and Circuit Tracing, the team examined the model’s internal activations in detail.

The paper offers an observable case:

In some unfaithful CoT cases, the reasoning text may not be the causal source of the answer. It may be a narrative woven to justify a predetermined answer. This is called motivated reasoning.

That means if you rely on CoT to ensure explainability in AI decisions, there is a trust boundary in the current architecture.

When “Explanation” Becomes Post-hoc Justification

In enterprise AI, what worries senior engineers and tech leads most is not just that a model gets the answer wrong. It’s that the model gets the answer right for absurd reasons, or gets it wrong with perfect-looking reasons.

Our intuition says: list the steps first, then reach the conclusion, and the process becomes transparent. But the paper shows that this intuition does not always hold inside a neural network.

In motivated-reasoning cases with human-suggested hints, the model may lock onto an answer first, then weave the logic backward to persuade you. But this is not the universal pattern. In faithful CoT cases without hints, the reasoning steps really do drive the answer.

That creates a practical limit: checking the CoT alone is not enough for causal verification. Because the CoT itself may be part of an unfaithful output. This is different from hallucination in the usual sense. It is not a random error. It is a highly persuasive narrative built with care.

The Core Mechanism: Why “Explanation” Can Fail

The key is not the prompt. It is where we cut into the black box.

This paper does not change model behavior through prompt engineering. It does it by directly observing and intervening on internal activations.

The key analogy: a voice recorder vs. a commit message

Imagine listening to an improvised talk.

  • The usual assumption: the speaker thinks while speaking, and the recorder, the CoT, faithfully captures the path of thought.
  • What the paper finds: the model behaves more like someone writing after the fact. It reaches for an answer first, then looks back and writes a reasoning process that sounds coherent. Like an engineer who ships the code first, then writes a tidy commit message afterward.

Where the intervention happens: The researchers did not edit the prompt. They used circuit tracing to intervene at specific internal layers and features.

They found this: When they suppressed early features tied to the final answer, the model lost confidence in that answer, and sometimes changed it. But when they suppressed features tied to the CoT steps, the model often kept the same answer, while the CoT became less coherent.

Key Insight: In motivated-reasoning cases, the causal chain runs from answer features to CoT features. More concretely, it comes from interactions among features like hint answer, divide-by-5, and say-8, not from an abstract “thinking process.”

What the Paper Shows: Two Testable Cases

The paper gives concrete experimental observations that directly challenge CoT reliability:

  1. Evidence of motivated reasoning: In certain complex reasoning tasks, when the model is given a strong hint, its CoT can drift completely away from the real computation path and align itself with the hint instead. This shows that CoT is not always faithful to the internal computation.
  2. Causal evidence from feature intervention: In a medical-diagnosis case, when researchers suppressed the internal feature tied to preeclampsia, the model not only changed its diagnosis, it also shifted its follow-up questioning strategy toward biliary disorders, cholecystitis, and decreased appetite. This shows that internal features do drive decisions, but those features do not map to the final CoT text in a stable causal way.

Your Judgment: Boundaries and Tradeoffs

Where it fits:

  • Fact retrieval and simple logic: CoT is only a supporting clue when the answer can be verified externally. The CoT itself should not be treated as the verification.
  • Low-risk scenarios with fast verification: In writing or coding assistance, if a human can review the output, CoT can serve as a communication aid. Correctness still needs to be confirmed by tests or human review. What users care about is whether the result is usable, not whether the reasoning process is causally faithful.

Where it does not fit well:

  • High-risk decisions: medical diagnosis, financial risk control, legal advice. In these domains, post-hoc rationalization in CoT can increase the risk of misplaced trust, so it needs external verification and audit support.
  • Compliance settings that require explainability: If regulation requires the decision path to be traceable, CoT should not stand alone as proof. It needs a stronger audit chain around it.

The tradeoff: If you want more faithful CoT, you may need to give up some model flexibility and intuition. For example, forcing the model to self-verify at every step usually adds tokens and verification steps, which can raise latency and cost. This is a classic accuracy vs. cost tradeoff.

A Toolmaker’s Observation

Anthropic’s engineering work on attribution graphs and circuit tracing gives us a way to open one corner of the black box. But in the current architecture, CoT offers user-facing narrative coherence, not engineering-grade causal proof.

CoT gives you a readable story. Engineering causality still needs attribution, intervention, or external verification. It is also worth noting that attribution graphs are still research tools. In production, external verification and audit chains are the more common path.

A Practical Rule You Can Take With You

You do not need to rebuild your whole system. But you can choose to keep CoT as a readable clue, and add attribution or external verification in high-risk flows.

This is not a simple right-or-wrong split. It is a tradeoff between risk level and verification cost. The next time you see a perfect CoT, ask one more question: is this CoT just a readable narrative, or is there external evidence behind it?

Before You Use This Paper to Challenge Your Architect, Get Clear on These Three Assumptions

  1. A hypothesis, not a final conclusion: Attribution graphs provide a mechanistic hypothesis, not absolute truth. Intervening inside the model may change its original behavior, which is an observer effect.
  2. Locked to one model version: The findings apply to Claude 3.5 Haiku. Other models, like GPT-4o or Llama 3, may have different internal mechanisms, so the result does not transfer directly.
  3. Dataset bias: Some conclusions come from test sets in specific domains, such as medicine and math. They may not fully represent model behavior across every open-domain task.

Sources