How Post-trained LLMs Recognize Their Own Voice

by
Table of contents

🌏 閱讀中文版本

When an agent reflects on itself, the entropy of its output distribution can drop by a factor of 3 to 4.

You built an agent. It starts correcting itself. Its answers get shorter. Its tone gets firmer. Users may experience that as being lectured, though the paper did not measure those user perceptions. What we can confirm so far is this: when a model recognizes that it is generating a response, its output distribution becomes markedly more concentrated. Behind this shift from passive prediction (Simulation) to active execution (Enaction) is a causal role for the model’s internal representation of Input Surprise.

The Core Question: How Does a Model Separate Generation from Input?

During pretraining, a language model is fundamentally a passive statistical predictor. Its goal is simple: given the preceding text, predict the next token. At this stage, the model has no reason to distinguish whether a passage came from a human or from itself. To the model, it is all just text.

After post-training steps such as supervised fine-tuning (SFT) and preference optimization (DPO), however, something fundamentally changes. A post-trained model is given a role, such as Assistant, and learns to respond to human expectations in conversation.

The key question this paper investigates is: we have long assumed that AI is simply continuing text, but it may already contain a form of self-monitoring. If a model can recognize that it is speaking, we may be able to influence its stability by intervening on that self-recognition. This remains a research hypothesis that needs further validation. The paper explicitly notes that it has not tested complex agent settings, where tool outputs, retrieved documents, and self-generated text are interleaved.

The Core Mechanism: Why Does This Happen?

To understand the mechanism, imagine a language model as a communication system.

During pretraining, the device is just a radio. It receives signals from every channel and uses statistical patterns to fill in the next sentence. It does not care who said the words or whether it is the transmitter.

After post-training, the device becomes a communicator with frequency lock. When it starts sending a message, it activates an internal self-review system. Its core metric is called Input Surprise.

Definition: Input Surprise is the model’s internal representation of how unexpected a recently observed input token is, based on its earlier prediction. More specifically, it is S =  − log P(actual input token). Entropy is the expected surprise over the full prediction distribution, so the two are not interchangeable.

When a model generates its own response, it is following an already established intent. Because the model established that intent itself, its expectation for what comes next is highly specific. In its internal neurons, that specificity appears as low surprise.

The key causal chain looks like this:

  1. Before generating its first token, the model has already established an internal topic intent for the response.
  2. As it continues generating text, the tokens align with the intent it established, so Input Surprise stays very low.

This is a naturally emerging low-entropy pattern. It is important to distinguish this from an internal intervention artificially applied through activation steering. What we observe is the model’s native response. The paper’s teleological explanation, that this behavior evolved to fulfill a role, has not been confirmed; for now, we can observe the phenomenon itself.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def measure_entropy(model, tokenizer, input_ids):
    # Disable training mode to ensure gradients are not updated
    model.eval()
    with torch.inference_mode():
        outputs = model(input_ids)
        # logits[:, t] corresponds to the prediction distribution for token t+1
        logits = outputs.logits[:, :-1, :] 
        input_ids_next = input_ids[:, 1:]
        
        # Calculate entropy at each token position: H = -sum(p * log(p))
        entropy = -torch.sum(probs * torch.log(probs + 1e-9), dim=-1)
        return entropy.mean()

# This function only demonstrates token-level entropy calculation for one sequence.
# To reproduce the on/off-policy comparison, keep the prompt and chat template identical,
# then compare entropy between self-generated text and externally provided text.

Before generating its first output token, a post-trained model already collapses uncertainty about the topic of the content it is about to produce. This is implicit self-recognition, rather than explicit instruction following.

Key Insight: Evidence Boundaries and Partial Causality

Before moving into practical applications, we need to clarify the boundaries of the evidence. This is an interpretable piece of neural mechanism, not a complete account of agent behavior.

  1. Correlation vs. causality: The paper shows a strong correlation between self-generated text and low entropy, and surprise steering has partial causal effects. The full formation mechanism is not yet clear, and surprise explains only part of the effect.

  2. Experimental scope:

    • Cross-model comparisons used 20 prompts.
    • Natural conversations used 30 topics.
    • Intent experiments used 8 prompt sets.
    • Surprise experiments also included chat and web contexts.

    The paper’s conclusions are based on these specific designs. Complex agent settings have not yet been validated.

  3. Implicit vs. explicit: Models can explicitly report whether their response was prefilled, but explicit recognition routes through neural circuits different from those used for implicit recognition. This suggests that “knowing what I said” and “sensing that something in the content is off” are two distinct cognitive tendencies.

The Result That Matters: A 3–4x Entropy Gap

The research team ran rigorous experiments across multiple model families and scales, producing several specific, falsifiable findings. These numbers are measured observations.

1. A 3–4x gap between on-policy and off-policy text

When a model generates its own response (on-policy), the entropy of its output distribution is 3 to 4 times lower than when it reads external text (off-policy). In other words, when the model is “speaking,” its range of candidate next tokens is much narrower than when it is “reading.”

This differs from the earlier 3–4x gap in an important way. Even when a model is simply reading, it can distinguish the source of text: entropy is lower when it reads text it generated itself than when it reads text generated by another model. This is known as the Self-advantage. The effect grows monotonically with model size. It is nearly absent in 2B-parameter models, while in 70B models the entropy difference can reach 0.1–0.4 nats.

3. The prefill experiment: intent conflict raises entropy

If we force-insert text that does not align with the model’s current intent, its output entropy rises significantly. This indicates that the model has a cached intention. When external input conflicts with that intention, the internal surprise mechanism is activated and the model becomes less certain.

Your Judgment: Applicability Boundaries and Practical Meaning

This finding offers direct signals for AI agent design, while its applicability boundaries are also clear.

Good fit:

  • Conversation systems with high stability requirements: If you want a model to preserve consistency of tone and reasoning during self-reflection or long-form generation, this low-entropy characteristic is worth considering. By keeping the model in an on-policy state, where it clearly recognizes that it is generating content, you can obtain more convergent and predictable output.
  • Safety monitoring and intervention: The research finds that KV Cache Patching, which overwrites the key-value cache for user tokens, can induce false-positive prefill detection or suppress true-positive detection. This means that operating on internal states can help us examine a model’s self-recognition mechanism, offering a new dimension for safety audits.

Less suitable situations:

  • Open-ended tasks requiring strong creativity: Low entropy means high certainty, but it may also make thinking more rigid. That remains a hypothesis. If a task requires the model to move beyond established frames and brainstorm boldly, relying too heavily on this self-recognition mechanism may reduce flexibility.
  • Very small models: The research shows that the Self-advantage effect is nearly absent at 2B scale. If your deployment uses models around 2B parameters, the current basis for judgment is insufficient to expect an effect of the same magnitude.

Decision framework:

You do not need to rebuild your entire agent architecture immediately. Confirming the role setup in a prompt may be enough to trigger this stability effect, and that is a testable hypothesis. When designing an agent, ask yourself three questions:

  1. Is the intent clear? If the model has not cached a clear intent before generation, the self-recognition mechanism may not operate effectively.
  2. Are external disruptions controllable? If input text frequently contradicts the model’s expectations, creating high surprise, its output may become less stable.

There is no causal dependency between internal entropy representations and output entropy itself, but surprise representations do have a meaningful regulatory effect on output entropy. This means the model’s internal “feeling” does not necessarily map directly to behavior, while a sense of unexpectedness can directly affect how certain that behavior is.

A careful observation on KV Cache Patching

This is a double-edged tool. Manipulating internal state is like replacing an engine mid-flight: theoretically feasible, but it requires a very high level of trust. The fragility of KV Cache means this mechanism could be maliciously used or triggered unexpectedly, though the paper does not establish an attack surface. Any discussion of security risk needs separate supporting evidence from security research. In real deployments, it is worth exploring other stability approaches rather than relying too heavily on internal-state manipulation.

Next Step: Observe the Model’s Internal State

For developers, this provides a new lens for observing a model’s internal state: comparing entropy can help validate the degree of a model’s self-recognition. If you are building an LLM-based agent, you can observe it along these dimensions:

  • Token entropy changes: During self-reflection or long-form generation, track fluctuations in output entropy. Limit the observable quantity to next-token distribution sharpness. Measuring correctness, behavioral stability, or self-recognition requires independent evaluations for each.
  • Role-marker activation: Observe whether explicitly assigning a role in the prompt, such as “Assistant,” leads the model to show lower entropy and stronger intent consistency.

Understanding this mechanism may become important for designing the next generation of stable agents. When a model starts to recognize itself, do you want its output to converge tightly around expectations, or retain flexibility at the boundary of exploration? That depends on the kind of certainty your system needs. In which setting do you want it to operate? That is for you to decide.

Sources