RLHF 的代價：InstructGPT 如何靠對齊而非規模，讓 1.3B 打敗 175B 的 GPT-3
175B 的 GPT-3 輸給了 1.3B 的 InstructGPT。這不是參數的勝利，是『聽話』的勝利。但聽話的代價，是你在部分任務上可能付出能力取捨。本文拆解 RLHF 機制與對齊稅，說明如何在可控性與通用性之間做出取捨。
Alignment Tax
The Cost of RLHF: Why 1.3B Beat 175B
GPT-3 at 175B lost to InstructGPT at 1.3B. This was not a win for parameter count. It was a win for instruction following. But that gain comes with a tradeoff: when you push a model to stay closer to human preference, you may also narrow its performance on some public NLP tasks. This article breaks down how RLHF works through SFT, reward models, and PPO, then looks at the alignment tax and what it means when you have to choose between controllability and broader capability.