mlsanityData to 14 Oct 2026

Regularized Emphatic Temporal-Difference Learning: Stability under Constant Stepsizes

arXiv 2609.19170 ↗First posted 14 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Emphatic temporal-difference learning (ETD) stabilizes the expected off-policy TD update and changes its projection geometry, but neither property determines constant-stepsize sampled dynamics. We construct an ergodic two-state counterexample in which the ETD mean map contracts while the sampled product has a positive top Lyapunov exponent. Regenerative-cycle analysis separates this sign from the infinite variance of the follow-on trace. We introduce regularized emphatic TD (RETD), a normalized first-order post-shock repair that leaves the trace and importance ratios unchanged, stores the emphatic TD signal in a leaky scalar state, and releases a delayed correction. RETD's raw equilibrium is an affine shift of the ETD equilibrium; single- and two-regularization readouts recover the ETD fixed point exactly. We prove almost-sure convergence for harmonic diminishing stepsizes and a conditional constant-stepsize moment-contraction result from a Markovian random-product bound. RETD has certified negative exponents on the two-state construction and one Baird point, whereas the positive Baird ETD sign remains numerical. Paired 10,000-run experiments validate both separations, fixed-point recovery, a nonmonotone stability region, and task dependence. RETD changes post-shock dynamics; it does not reduce the shared follow-on-trace variance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

19 of 31 references.

References · 19

  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  3. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  4. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  5. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  6. 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noisecited by 24
  7. 2021On the Stability of Random Matrix Product with Markovian Noise: Application to Linear Stochastic Approximation and TD Learningcited by 23
  8. 2015On Convergence of Emphatic Temporal-Difference Learningcited by 21
  9. 2023Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stabilitycited by 20
  10. 2015Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysiscited by 17
  11. 2020Multiplicative noise and heavy tails in stochastic optimizationcited by 17
  12. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12
7 more
  1. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  2. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8
  3. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  4. 2015Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsizecited by 4
  5. 2021PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Methodcited by 2
  6. 2025Bellman Error Centeringcited by 2
  7. 2026A Single Stepsize Suffices for Unprojected Linear TD(0): Simultaneous Robust and Fast Rates via Polyak--Ruppert Averagingcited by 2

Built on by

Not cited yet.