Adaptive Multi-Horizon RL: Engineering Tradeoffs of State-Dependent Gating and Fixed γ

by
Table of contents

🌏 閱讀中文版本

Results Reported in the Paper and Requirements for Reproduction

In continual learning scenarios for Reinforcement Learning (RL), agents face a central tradeoff: the discount factor, γ, determines how much future reward weighs on the current decision. A single fixed γ works well when an environment’s temporal structure is stable. But when tasks switch frequently and causal-chain lengths change substantially, performance can decline.

Based on the research team’s paper, Adaptive Multi-Horizon Reinforcement Learning, this article explains how a state-dependent gating network lets an agent dynamically weight critic networks with multiple γ values.

Observable result: In multi-task switching experiments in MiniGrid, the architecture automatically favored the short temporal perspective during the Foraging task during the first 4k episodes, then adjusted its weights as tasks changed. Final episode returns stayed stable between 36 and 40, without a clear decline in performance.

Requirements for reproduction:

  • The environment has observable state features that can distinguish the needs of different temporal structures.
  • Compute resources can support parallel execution of multiple critic networks and updates to the gating weights.
  • The goal is to reduce hyperparameter-tuning overhead, rather than optimize a single task to its absolute limit.

Steps: Implementing a State-Dependent Multi-Horizon Mixer

To reproduce this mechanism, build critic networks with different γ values and introduce a gating network for dynamic weighting. Here are the core steps and execution constraints.

Step 1: Initialize Multi-Horizon Critic Networks

Maintain a set of critic networks, Qγi(s,a), with different discount factors γi. Each network learns value estimates over a different time horizon.

  • Prerequisite: Define the sampling range for γ, for example from 0.5 for a short horizon to 0.99 for a long horizon.
  • Check: Confirm that each critic can converge independently for its assigned γ value, avoiding conflicts in initial weights.
  • Stop condition: Continue monitoring each critic’s loss and Q-value distribution. Initialization is complete only when loss no longer rises continuously or oscillates with increasing amplitude, and each Q-value distribution no longer shows clear drift.
  • Recovery path: If a critic does not converge, with loss continuing to rise or oscillate, try lowering that critic’s learning rate or check whether an extreme γi is causing vanishing gradients.

Initialization records should retain each critic’s independent loss and Q-value distribution rather than showing only aggregate results. This boundary matters: a stable overall curve can still conceal continued drift in one temporal perspective. If any critic is not yet stable, retain its most recent model state that did not deteriorate, verify its inputs, update rule, and discount-factor settings, then bring it back into the mixer.

Step 2: Build a State-Dependent Gating Network

Design a small neural network as a gate. It takes the current state s as input and outputs a weight distribution, wi(s), across critics. The final mixed value function is:

Qmix(s,a) = ∑iwi(s)Qγi(s,a)

  • Prerequisite: State features need enough information density to distinguish the temporal-structure requirements of the current task.
  • Check: Verify that the gate output is a valid probability distribution whose weights sum to 1.
  • Stop condition: Treat the weight allocation as stable only when identical or equivalent state inputs consistently produce repeatable allocation patterns, and weights no longer oscillate irregularly with increasing amplitude or remain concentrated on one critic over time.
  • Recovery path: If weights converge on a single critic without differentiation, check whether state features are sufficiently informative or adjust the gating network’s learning rate.

Step 3: Run Undiscounted Expected SARSA Updates

To optimize the gating network and mixed value, the system uses an undiscounted (γ = 1) Expected SARSA update rule. The objective uses squared TD-error loss:

$$ L = \frac{1}{2} \delta^2 $$

The TD error (δ) is:

δ = rt + 1 + ∑a′π(a′|st + 1)Qmix(st + 1,a′) − Qmix(st,at)

  • Prerequisite: The policy π(s) is determined by the mixed value: π(s) = arg maxaG(s,a), where G(s,a) is the weighted value.
  • Check: Observe whether TD error moves from continuous growth or sharp oscillation toward stability, while confirming that gating weights no longer shift widely and irregularly.
  • Stop condition: When TD error no longer grows continuously, its fluctuations no longer expand, and gate weights show repeatable allocations for the same state, stop this update round and move to task-switching validation.
  • Recovery path: If TD error continues to grow or oscillation intensifies, pause updates to the gating network and restore the most recent model state where TD error had not deteriorated. Then verify the TD target, policy output, and Qmix against the same update equation before resuming joint updates once the error is no longer expanding.

This step’s stop condition cannot rely on declining loss alone. The mixer’s purpose is to map states to different temporal perspectives, so identical state inputs must also produce repeatable weight patterns. If TD error becomes stable while weights still jump irregularly, the update is not complete. If weights are stable while TD error continues to deteriorate, fixed output should not be treated as convergence.

Failure Cases and Observations: Different Discount Rates Change Behavior

During implementation, distinguish between “the mechanism is working” and “the parameters are sensitive.” The paper notes that when γ < 0.5, the agent tends to choose the “left branch”; when γ > 0.5, it switches to the “right branch.” This shows that policy is highly sensitive to the discount rate, but it does not directly establish that the gating network adapts better than a fixed γ.

Signals to watch:

  • Weights do not differentiate: The gating network consistently concentrates weights on one γ value, reducing the system to a fixed-discount model. This usually occurs when state features cannot distinguish task differences.
  • Large return fluctuations: Early in task switching, if gating weights adjust too slowly or too quickly, short-term returns may decline substantially. Check the stability of TD-error updates in this case.

Diagnosis can proceed in two layers. First, confirm that each fixed-γ critic exhibits the behavioral differences described in the paper. If not, the problem lies in baseline value estimation rather than the gate. Only then inspect whether weights move during task switching, and whether that movement appears alongside return performance. This order narrows the debugging blast radius and avoids repeatedly tuning the gate before critics establish distinguishable perspectives.

Validation: Compare Weights and Returns with Figures 6–9

To confirm that the system captures “temporal perspective,” validate it through the following indicators:

  1. Monitor the gating-weight distribution: Record how the gating network’s weights for each γ change across task phases, such as switching from Foraging to Goal Reaching. According to Figures 6–8, Foraging has an optimal effective horizon of 16–32 steps, Goal Reaching 32–512 steps, and Four Rooms 128 steps. If weights shift with the task, the mechanism is operating as intended.
  2. Return stability: As shown in Figure 9a, a successful implementation keeps episode returns between 36 and 40, without a clear decline during continual learning.
  3. Behavioral consistency: Observe the agent’s behavioral branch choices, left or right, under different γ settings and confirm that they align with the expected temporal perspective.

For validation, use a checklist that moves from local signals to the overall result: first check whether fixed γ values create observable behavioral differences; then verify each task’s effective horizon; next observe whether gating weights shift after task changes; and finally compare episode returns. These four signals serve different roles and cannot replace one another. Consistent branch behavior validates discount-rate sensitivity; weight shifts validate that the gate uses state to adjust the mixed result; and returns within the paper’s reported range check the overall reproduction outcome.

If fixed critics reproduce behavioral differences but gating weights do not shift between tasks, focus on state representation or gating updates. If weights have shifted but returns still fluctuate substantially, return to the update path for TD error and mixed value. This framework does not treat one visually smooth curve as sufficient evidence. It requires mechanism signals and outcome signals to align.

Applicability Boundaries and Engineering Tradeoffs

This method is not a cure-all; its benefits depend on the specific setting. Here is an engineering tradeoff analysis based on the paper’s results.

Suitable Scenarios

  • Continual-learning environments: Tasks switch frequently, and different tasks have distinctly different temporal structures, such as the contrast between Foraging and Goal Reaching.
  • Parameter-efficiency sensitivity: Applications that aim to reduce hyperparameter-tuning overhead and let the system adapt to environmental change automatically.

Tradeoff Analysis

  • Higher compute requirements: Maintaining multiple critic networks and computing gating weights in real time increases training and inference overhead. This exchanges additional compute for hyperparameter stability; its marginal benefit depends on task-switching frequency and the human cost of tuning.
  • Dependence on state features: Gating-network performance depends heavily on the quality of state features. If state representations cannot distinguish the needs of different temporal perspectives, the system may not weight them correctly.

Decision Rules

When evaluating whether to adopt this method, consider these criteria:

  1. Task-switching frequency: If tasks switch often and their temporal structures differ significantly, this method becomes a stronger candidate. Otherwise, a fixed-γ baseline is a more suitable place to begin validation.
  2. Compute resources: Ensure sufficient capacity for parallel computation across multiple critic networks.
  3. State observability: Confirm that state features reflect the current task’s temporal requirements. If information density is insufficient, improving state representation is worth considering before introducing a more complex gate.

Limitations and Next Validation Steps

Experimental Scope and Transfer Boundaries

Current empirical results mainly come from environments with discrete action spaces, including MiniGrid. Performance in continuous action spaces or with more complex visual inputs needs further validation. The gating network’s effectiveness also depends on states correctly identifying temporal-perspective requirements; if environmental dynamics are highly complex, it may face convergence difficulties.

Next Validation Steps

For teams working on dynamic task switching, the following dimensions can help assess whether a system has captured “temporal perspective”:

  • Monitor the gating-weight distribution: Record the trend in gating weights for each γ across task phases, such as switching from Foraging to Goal Reaching. This is an important indicator of whether the system has learned to adjust planning depth to the context.
  • Validate weight-response speed: When tasks switch, record the slope of change in gating weights. If a shallow slope coincides with lower performance, check whether the discriminative power of state features is sufficient to support decisions across multiple time scales.

The paper reports that, in MiniGrid task-switching experiments, episode returns remained between 36 and 40, and the mixer assigned most of its weight during the first 4k Foraging episodes to discount factors corresponding to short effective horizons. The next bounded and safe validation is to reproduce these two observable results within the same task-switching scope: confirm that weights move from Foraging’s 16–32-step effective horizon to the horizons required by subsequent tasks, while verifying that returns remain within the range reported in the paper. Once this comparison is complete, you can decide whether extending validation to continuous actions or more complex visual inputs is worth considering.

Sources