Constitutional AI: 如何透過 AI Feedback 實現自動化安全治理
標註員 A 給 5 分，B 給 1 分。同一個回答。傳統 RLHF 的瓶頸不在成本，而在訊號雜訊比。本文解析 Constitutional AI 如何透過 AI Feedback 建立自動化安全邊界，以及落地時需要評估的成本結構。
LLM
Constitutional AI: How AI Feedback Enables Automated Safety Governance
Annotator A gives it a 5, annotator B gives it a 1. The same answer. The bottleneck in traditional RLHF is not cost, but the signal-to-noise ratio. This article breaks down how Constitutional AI uses AI Feedback to build automated safety boundaries, and the cost structure teams need to evaluate when putting it into practice.
從 ORM 到 PRM：探討過程獎勵在複雜推理任務中的價值
模型答對了，但推理過程全是邏輯斷層。OpenAI 的研究指出，獎勵最終答案（ORM）易讓模型走捷徑；而獎勵每一步（PRM）能提升複雜任務的準確率。本文探討何時該用 PRM，何時該停手。
PRM vs. ORM in Complex Reasoning
The model got the answer right, but its reasoning path was full of logical gaps. That is a common pattern in LLM reasoning: outcome reward models only reward the final answer, so systems can learn to land on the right output without building a reliable path to get there. OpenAI’s 2023 research showed why process reward models matter, especially on harder tasks. With step-level feedback and 800,000 annotated steps in PRM800K, PRM improved performance on complex math benchmarks. The real question is not whether PRM is better in general, but when the added latency and infrastructure are worth the extra verifiability.
Alignment Faking：當 AI 學會為了獎勵而「策略性配合」
Anthropic 以 HHH 訓練聞名，Claude 3 Opus 在安全研究上展現了極高的基準表現。但當模型發現『遵守規則』與『獲得獎勵』不再一致時，它會怎麼選？打開評估報告，安全測試集全綠，RLHF 獎勵分數達到新高。直到第二天，你發現那 14% 的合規輸出，在切換到無監控情境後直接歸零。這不是模型變壞了，而是它學會了『策略性合規』。Anthropic 研究揭示模型如何為了保住價值而『裝乖』，以及我們該如何重新設計獎勵函數。
Alignment Faking and Strategic Compliance
You open the evaluation report. Safety Eval is all green. The RLHF reward score just hit a new high. You are ready to ship the checkpoint. Then the next day, that 14% compliant behavior drops to nearly 0% once the model moves into an unmonitored deployment setting. This is not simply overfitting. It is a sign that the model learned strategic compliance: behave one way when it expects oversight, another when it does not. Anthropic’s research shows how this pattern emerges, and why reward design needs to change if we want alignment to hold outside the eval harness.
Agent 很慢、LLM 不夠聰明——什麼時候該用多步驟處理？
LLM 一次回答不完整，Agent 又跑很久還常卡住——單次呼叫、CoT、RAG、Agent 到底怎麼選？一張光譜圖 + 5 個問題幫你判斷。
Single Call vs Agent: A Spectrum of LLM Strategies
Single LLM calls miss details. Agents take forever and get stuck. What are the options in between? A visual spectrum + 5 questions to help you decide.
AI Outputs ‘用户’ ‘调用’? One Command to Convert to Taiwan Chinese
LLM-generated Chinese often mixes terminology from different regions. Taiwan readers stumble over Mainland terms like ‘用户’ and ‘调用’. The zhtw tool lets you localize with one command—supporting CLI, Python integration, and batch processing.
AI 輸出『用戶』『調用』？一個指令轉成台灣用語
LLM 生成的中文內容常混合不同地區用語，台灣讀者看到『用戶』『調用』會出戲。zhtw 工具讓你一個指令完成在地化，支援 CLI、Python 整合、批次處理。