長程決策中的成對偏好強化學習：Markov Decision Contest 的機制與邊界
長程決策中，標量獎勵難以精確定義，而成對偏好方法在理論保證上存在缺口。研究提出的 Markov Decision Contest 模型將求解複雜度證明為 P 類，並提出 HPI 與 HPI-Clip 算法。實測顯示近似算法在多個高維任務中學習效率優於 SPPO，為長程成對偏好決策提供新的邊界與取捨。
PPO
Pairwise Preference Reinforcement Learning in Long-Horizon Decisions: The Mechanisms and Boundaries of Markov Decision Contest
In long-horizon decision-making, scalar rewards are difficult to define precisely, while pairwise preference methods have gaps in their theoretical guarantees. Markov Decision Contest proves that its exact solution is in P and introduces the HPI and HPI-Clip algorithms. Experiments across seven high-dimensional control tasks show that the approximate HPI-Clip method learns more efficiently than SPPO. The result defines a new set of boundaries and trade-offs for long-horizon decisions based on pairwise preferences.
RLHF 的代價：InstructGPT 如何靠對齊而非規模，讓 1.3B 打敗 175B 的 GPT-3
175B 的 GPT-3 輸給了 1.3B 的 InstructGPT。這不是參數的勝利，是『聽話』的勝利。但聽話的代價，是你在部分任務上可能付出能力取捨。本文拆解 RLHF 機制與對齊稅，說明如何在可控性與通用性之間做出取捨。
The Cost of RLHF: Why 1.3B Beat 175B
GPT-3 at 175B lost to InstructGPT at 1.3B. This was not a win for parameter count. It was a win for instruction following. But that gain comes with a tradeoff: when you push a model to stay closer to human preference, you may also narrow its performance on some public NLP tasks. This article breaks down how RLHF works through SFT, reward models, and PPO, then looks at the alignment tax and what it means when you have to choose between controllability and broader capability.