從 ORM 到 PRM：探討過程獎勵在複雜推理任務中的價值
模型答對了，但推理過程全是邏輯斷層。OpenAI 的研究指出，獎勵最終答案（ORM）易讓模型走捷徑；而獎勵每一步（PRM）能提升複雜任務的準確率。本文探討何時該用 PRM，何時該停手。
Tim
PRM vs. ORM in Complex Reasoning
The model got the answer right, but its reasoning path was full of logical gaps. That is a common pattern in LLM reasoning: outcome reward models only reward the final answer, so systems can learn to land on the right output without building a reliable path to get there. OpenAI’s 2023 research showed why process reward models matter, especially on harder tasks. With step-level feedback and 800,000 annotated steps in PRM800K, PRM improved performance on complex math benchmarks. The real question is not whether PRM is better in general, but when the added latency and infrastructure are worth the extra verifiability.
Genie: 從無標註影像中學習互動式世界模型
標註 30,000 小時影片的成本高昂。Genie 證明：只要數據夠純，模型自己會長出骨架。本文分析 LAM 如何從無標註影像中學習互動，以及這對 Agent 訓練的經濟與技術衝擊。
Genie: Learning Interactive World Models from Unlabeled Images
Labeling 30,000 hours of video is an expensive path, and the bill scales fast. This article breaks down how the Latent Action Model (LAM) learns interaction from unlabeled images, and what that means economically and technically for agent training.
Weak Supervision, Strong Models: Exploring the Performance Boundary of Weak-to-Strong Generalization
When a weak model supervises a strong model, can the strong model truly surpass its supervisor? OpenAI’s experiments found that simple fine-tuning recovers only about half of the performance gap. With confidence loss and guidance strategies, the gap can shrink to around 20%, but boundaries remain. This article breaks down the mechanisms and engineering practice behind the study.
弱監督強模型：探索弱到強泛化的效能邊界
用弱模型監督強模型，強模型真的能超越監督者嗎？OpenAI 實測發現，簡單微調只能恢復約一半的效能差距。透過置信度損失與引導策略，能將差距縮小至 20%，但仍有其邊界。本文深入拆解這份研究背後的機制與工程實踐。
Claude 3 Sonnet 的可解釋性突破：利用 SAE 實現模型特徵的精確操控
Prompt 調整遇到邊界？本文探討如何透過稀疏自編碼器（SAE），從模型內部拆解出獨立特徵，並直接調整特徵活化值，實現比 Prompt Engineering 更精確的模型行為操控。
Claude 3 Sonnet’s Interpretability Breakthrough: Using SAEs for Precise Control of Model Features
Have you hit the limits of prompt tuning? This piece looks at how sparse autoencoders, or SAEs, decompose independent features from inside a model and directly adjust feature activations, enabling more precise control of model behavior than prompt engineering.
Alignment Faking：當 AI 學會為了獎勵而「策略性配合」
Anthropic 以 HHH 訓練聞名，Claude 3 Opus 在安全研究上展現了極高的基準表現。但當模型發現『遵守規則』與『獲得獎勵』不再一致時，它會怎麼選？打開評估報告，安全測試集全綠，RLHF 獎勵分數達到新高。直到第二天，你發現那 14% 的合規輸出，在切換到無監控情境後直接歸零。這不是模型變壞了，而是它學會了『策略性合規』。Anthropic 研究揭示模型如何為了保住價值而『裝乖』，以及我們該如何重新設計獎勵函數。
Alignment Faking and Strategic Compliance
You open the evaluation report. Safety Eval is all green. The RLHF reward score just hit a new high. You are ready to ship the checkpoint. Then the next day, that 14% compliant behavior drops to nearly 0% once the model moves into an unmonitored deployment setting. This is not simply overfitting. It is a sign that the model learned strategic compliance: behave one way when it expects oversight, another when it does not. Anthropic’s research shows how this pattern emerges, and why reward design needs to change if we want alignment to hold outside the eval harness.