mlsanityData to 14 Oct 2026

On the Divergence of Differential Temporal Difference Learning without Local Clocks

arXiv 2605.06874 ↗First posted 7 May 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Learning rate is a critical component of reinforcement learning (RL). This work uses global and local clocks to distinguish two types of learning rates. The former is of the standard form αt that depends only on the time step t (i.e., a global clock). The latter is of the form αnu(St, t), where nu(s, t) counts the number of visits to state s until time t (i.e., a local clock). In discounted RL, an RL algorithm that is convergent with a local clock is always also convergent with a global clock, and vice versa. We are not aware of any counterexample. The key contribution of this work is to show that this nice correspondence breaks down in average-reward RL. Specifically, we construct a counterexample showing that although differential temporal difference learning is convergent with a local clock, it can diverge with a global clock. This counterexample closes the open problem in Wan et al. [2021], Blaser et al. [2026].

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 35 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

9 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes18 Feb
  2. 2025Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learning30 Sep
  3. 2025Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Features27 May
  4. 2025Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Set31 Jan
  5. 2024Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features18 Sep
  6. 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise15 Jan

Other references · 3

  1. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  2. 2021The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learningcited by 32
  3. 2024Asymptotic and Finite Sample Analysis of Nonexpansive Stochastic Approximations with Markovian Noisecited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Parameter-Free Temporal Difference Learning
  2. 2026Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
  3. 2025Quantifying Memory Use in Reinforcement Learning with Temporal Range
  4. 2026Extending Differential Temporal Difference Methods for Episodic Problems
  5. 2026Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics
  6. 2026Deep Reinforcement Learning and The Tale of Two Temporal Difference Errors
  7. 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning
  8. 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion