Pairwise Preference Reinforcement Learning in Long-Horizon Decisions: The Mechanisms and Boundaries of Markov Decision Contest

by

In long-horizon decision-making, scalar rewards are difficult to define precisely, while pairwise preference methods have gaps in their theoretical guarantees. Markov Decision Contest proves that its exact solution is in P and introduces the HPI and HPI-Clip algorithms. Experiments across seven high-dimensional control tasks show that the approximate HPI-Clip method learns more efficiently than SPPO. The result defines a new set of boundaries and trade-offs for long-horizon decisions based on pairwise preferences.

長程決策中的成對偏好強化學習：Markov Decision Contest 的機制與邊界

by

長程決策中，標量獎勵難以精確定義，而成對偏好方法在理論保證上存在缺口。研究提出的 Markov Decision Contest 模型將求解複雜度證明為 P 類，並提出 HPI 與 HPI-Clip 算法。實測顯示近似算法在多個高維任務中學習效率優於 SPPO，為長程成對偏好決策提供新的邊界與取捨。

Decoupled DiLoCo: How Decoupling Improves Resilience in Large-Scale Distributed Pre-training

by

Large language model pre-training can pause when hardware latency or node outages propagate through strict synchronization points. Decoupled DiLoCo uses independent Learners and asynchronous integration, allowing training to continue in simulated environments with millions of chips. Under the paper’s specified fault model, it reports zero global downtime and sustains 88% Goodput. The result is a more resilient architectural option for distributed training.

When Does Stability Beat Plasticity in Gradually Non-Stationary Environments?

by

In gradually non-stationary environments, the main cause of reinforcement learning performance decline is instability caused by over-adaptation. This study finds that applying synaptic consolidation to multi-timescale Successor Features (SFs) can improve performance, but at the cost of computational latency and dependence on SGD optimizers.

Slot Machines: How LLMs Track Multiple Entities

by

Anthropic’s Slot Machines study shows that, on specific complex syntactic binding tasks, the models tested perform close to random chance while still maintaining an orderly internal representation through orthogonal current-entity and prior-entity slots. This article explains how that mechanism shapes expectations for model capability, and why understanding these boundaries offers more value for risk control than forcing a model to handle complex bindings in a single position.

Slot Machines：LLM 如何追蹤多個實體

by

Anthropic 研究《Slot Machines》顯示，受測模型在特定複雜句法綁定任務上表現接近隨機，但內部仍透過正交的 current-entity 與 prior-entity 槽位維持資訊秩序。本文解析此機制如何影響我們對模型能力的預期，並指出理解這些邊界比強迫模型處理複雜綁定更具風險控制價值。