The Cost of RLHF: Why 1.3B Beat 175B

by
Table of contents

🌏 閱讀中文版本

In an architecture meeting, the PM forwarded a note from the client: “Answers in this scenario need to be highly accurate and stay tightly within the instruction.”

The engineers looking at the screen already knew the cost behind that sentence. If you want a model to converge tightly around what you expect, you have to lower the probability of outputs that fall outside that preference. That can reduce hallucinations. It can also narrow the model’s range on other tasks.

OpenAI’s InstructGPT experiment exposed a counterintuitive result: in human preference evaluations over OpenAI API prompt distributions, InstructGPT at 1.3B outperformed GPT-3 at 175B.

When we push a model to “follow instructions,” some public NLP tasks may show regression, and that needs to be confirmed through downstream evaluation. That is the Alignment Tax.

175B lost to 1.3B.

This was not a win for parameter count. It was a win for obedience.

But the price of obedience is a capability tradeoff across some task distributions.

The Core Tradeoff

GPT-3 is a powerful continuation machine. You give it an opening, and it continues based on probability. It does not naturally answer your question.

That creates two pain points:

  1. Limited controllability: In instruction-following scenarios, the model tends to continue rather than answer, so you need extra prompt design to steer it.
  2. Adoption friction: In real applications, users want answers, not story continuation.

The engineering value of InstructGPT is that it brings human preference into the training loop. But the team also observed a tradeoff: alignment has a cost.

The Core Mechanism: Preference Optimization for Output Policy

RLHF has three stages: SFT, RM, and PPO. Let’s explain them from a systems architecture perspective.

1. SFT (Supervised Fine-Tuning): Redrawing the Decision Boundary

SFT is not teaching knowledge. It is redrawing the decision boundary. We give the model thousands of question-answer pairs and ask it to imitate them.

Where the intervention happens: This is not a prompt trick. This changes the model’s weights so that when it sees an instruction, the activated probability distribution leans toward answering instead of continuing.

2. RM (Reward Model): Preference Scoring

We train a separate reward model to score the model’s outputs. That model is trained on human preference: if humans prefer answer A over B, the RM gives A a higher score.

Analogy: The RM is not the teacher. It is the scoring function. It does not modify the model directly. It tells the model: “This gets rewarded. That gets penalized.”

3. PPO (Proximal Policy Optimization): Running in a Narrow Hallway

It sounds simple. But here’s the thing:

PPO adjusts the model’s policy based on feedback from the RM. The key is the KL penalty: it keeps the model from drifting too far from the original distribution.

The KL constraint is a divergence penalty. It makes sure that when the model updates its policy to better match human preference, its output distribution does not move too far from the original pretrained base model.

RLHF optimizes output policy under a preference model. That can improve controllability, but it does not automatically improve factuality. It is like running in a narrow hallway, with the walls moving closer. The KL constraint limits drift from the base model. Capability changes need to be observed through downstream evaluation, not inferred directly from weight space.

There may be a tradeoff between controllability and performance on some tasks. You gain tighter control on specific instructions, but you may pay for it on some general benchmarks. That is the cost of convergence.

What Actually Matters

Key Insight: InstructGPT showed that with RLHF, a 1.3B model could outperform GPT-3 175B in instruction-following ability.

  1. Parameter count is not the only driver of instruction-following preference: Human labelers preferred InstructGPT 1.3B over GPT-3 175B. That means the gain from alignment outweighed the scale gap in this API prompt preference evaluation.
  2. The alignment tax: Based on the InstructGPT paper’s tables and results, InstructGPT performed better on customer tasks, but worse on public datasets like SQuAD, DROP, and HellaSwag. That suggests the model became more specialized, but also more uneven. This was regression on some public datasets, not a broad decline everywhere.
  3. A mixed-data mitigation: During RL fine-tuning, mixing in a small portion of the original pretraining data can reduce the alignment tax. It is like teaching manners without losing the basics.

Your Judgment

Where It Fits

  • A good fit: Scenarios that need strong instruction following and consistent replies, such as customer support or information lookup. Safety evaluation still matters.
  • Needs additional evaluation: Complex reasoning, open-ended research, and creative generation. The question is whether post-alignment evaluation shows those capabilities are still there.

The Tradeoff

  • Alignment vs. capability: Alignment strength, data mixing, and KL constraints all affect general-task performance. As alignment gets stronger, some general tasks may need to be reevaluated for regression. You trade some edge-case breadth for controllability in your core scenario. This is not a bad thing. It is pricing.
  • Data quality vs. quantity: In SFT, data quality matters more than volume. SFT data quality affects generalization, but that still needs to be confirmed through validation sets and preference evaluations.

Practical Intent

  • It is worth confirming whether your goal is usability, controllability, or raw capability before deciding how much to invest in scale.
  • Monitor the alignment tax: Regularly evaluate the model on general tasks so you do not over-optimize for one specific workflow.

Limits

InstructGPT can still produce harmful content, and it can still be unstable on unseen instructions. The alignment tax is still an area that can be improved. This is not the end state. It is an engineering compromise for where the field is now. Before you invest resources, make sure your scenario can absorb that cost.

Sources

This is not a single answer. It is a choice about task boundaries and cost structure.

For your model’s task, what kind of controllability is worth paying for?