Reinforcement Learning

模型答對了，但推理過程全是邏輯斷層。OpenAI 的研究指出，獎勵最終答案（ORM）易讓模型走捷徑；而獎勵每一步（PRM）能提升複雜任務的準確率。本文探討何時該用 PRM，何時該停手。

The model got the answer right, but its reasoning path was full of logical gaps. That is a common pattern in LLM reasoning: outcome reward models only reward the final answer, so systems can learn to land on the right output without building a reliable path to get there. OpenAI’s 2023 research showed why process reward models matter, especially on harder tasks. With step-level feedback and 800,000 annotated steps in PRM800K, PRM improved performance on complex math benchmarks. The real question is not whether PRM is better in general, but when the added latency and infrastructure are worth the extra verifiability.