RLHF 的代價：InstructGPT 如何靠對齊而非規模，讓 1.3B 打敗 175B 的 GPT-3
175B 的 GPT-3 輸給了 1.3B 的 InstructGPT。這不是參數的勝利，是『聽話』的勝利。但聽話的代價，是你在部分任務上可能付出能力取捨。本文拆解 RLHF 機制與對齊稅，說明如何在可控性與通用性之間做出取捨。
Technical
IT industry technical articles covering AWS, Azure, Java, Python, SQL, DevOps, and more.
The Cost of RLHF: Why 1.3B Beat 175B
GPT-3 at 175B lost to InstructGPT at 1.3B. This was not a win for parameter count. It was a win for instruction following. But that gain comes with a tradeoff: when you push a model to stay closer to human preference, you may also narrow its performance on some public NLP tasks. This article breaks down how RLHF works through SFT, reward models, and PPO, then looks at the alignment tax and what it means when you have to choose between controllability and broader capability.
從 ORM 到 PRM：探討過程獎勵在複雜推理任務中的價值
模型答對了，但推理過程全是邏輯斷層。OpenAI 的研究指出，獎勵最終答案（ORM）易讓模型走捷徑；而獎勵每一步（PRM）能提升複雜任務的準確率。本文探討何時該用 PRM，何時該停手。
PRM vs. ORM in Complex Reasoning
The model got the answer right, but its reasoning path was full of logical gaps. That is a common pattern in LLM reasoning: outcome reward models only reward the final answer, so systems can learn to land on the right output without building a reliable path to get there. OpenAI’s 2023 research showed why process reward models matter, especially on harder tasks. With step-level feedback and 800,000 annotated steps in PRM800K, PRM improved performance on complex math benchmarks. The real question is not whether PRM is better in general, but when the added latency and infrastructure are worth the extra verifiability.
Genie: 從無標註影像中學習互動式世界模型
標註 30,000 小時影片的成本高昂。Genie 證明：只要數據夠純，模型自己會長出骨架。本文分析 LAM 如何從無標註影像中學習互動，以及這對 Agent 訓練的經濟與技術衝擊。
Genie: Learning Interactive World Models from Unlabeled Images
Labeling 30,000 hours of video is an expensive path, and the bill scales fast. This article breaks down how the Latent Action Model (LAM) learns interaction from unlabeled images, and what that means economically and technically for agent training.
弱監督強模型：探索弱到強泛化的效能邊界
用弱模型監督強模型，強模型真的能超越監督者嗎？OpenAI 實測發現，簡單微調只能恢復約一半的效能差距。透過置信度損失與引導策略，能將差距縮小至 20%，但仍有其邊界。本文深入拆解這份研究背後的機制與工程實踐。
Weak Supervision, Strong Models: Exploring the Performance Boundary of Weak-to-Strong Generalization
When a weak model supervises a strong model, can the strong model truly surpass its supervisor? OpenAI’s experiments found that simple fine-tuning recovers only about half of the performance gap. With confidence loss and guidance strategies, the gap can shrink to around 20%, but boundaries remain. This article breaks down the mechanisms and engineering practice behind the study.
Claude 3 Sonnet 的可解釋性突破：利用 SAE 實現模型特徵的精確操控
Prompt 調整遇到邊界？本文探討如何透過稀疏自編碼器（SAE），從模型內部拆解出獨立特徵，並直接調整特徵活化值，實現比 Prompt Engineering 更精確的模型行為操控。
Claude 3 Sonnet’s Interpretability Breakthrough: Using SAEs for Precise Control of Model Features
Have you hit the limits of prompt tuning? This piece looks at how sparse autoencoders, or SAEs, decompose independent features from inside a model and directly adjust feature activations, enabling more precise control of model behavior than prompt engineering.