mlsanityData to 14 Oct 2026

Extending Differential Temporal Difference Methods for Episodic Problems

arXiv 2605.04368 ↗First posted 6 May 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Differential temporal difference (TD) methods are value-based reinforcement learning algorithms that have been proposed for infinite-horizon problems. They rely on reward centering, where each reward is centered by the average reward. This keeps the return bounded and removes a value function's state-independent offset. However, reward centering can alter the optimal policy in episodic problems, limiting its applicability. Motivated by recent works that emphasize the role of normalization in streaming deep reinforcement learning, we study reward centering in episodic problems and propose a generalization of differential TD. We prove that this generalization maintains the ordering of policies in the presence of termination, and thus extends differential TD to episodic problems. We show equivalence with a form of linear TD, thereby inheriting theoretical guarantees that have been shown for those algorithms. We then extend several streaming reinforcement learning algorithms to their differential counterparts. Across a range of base algorithms and environments, we empirically validate that reward centering can improve sample efficiency in episodic problems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 36 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

13 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers22 Nov

Other references · 12

  1. 2016Layer Normalizationcited by 5,669
  2. 2018DeepMind Control Suitecited by 956
  3. 2023Understanding plasticity in neural networkscited by 174
  4. 2016Learning values across many orders of magnitudecited by 85
  5. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  6. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  7. 2016Unifying task specification in reinforcement learningcited by 41
  8. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  9. 2024Streaming Deep Reinforcement Learning Finally Workscited by 25
  10. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  11. 2024Reward Centeringcited by 16
  12. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
  2. 2025Implicit Updates for Average-Reward Temporal Difference Learning
  3. 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion
  4. 2026Towards Parameter-Free Temporal Difference Learning
  5. 2025An Empirical Study of Deep Reinforcement Learning in Continuing Tasks
  6. 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning
  7. 2024Multi-State TD Target for Model-Free Reinforcement Learning
  8. 2026Deep Reinforcement Learning and The Tale of Two Temporal Difference Errors