mlsanityData to 14 Oct 2026

Multi-State TD Target for Model-Free Reinforcement Learning

arXiv 2405.16522 ↗First posted 26 May 2024cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Temporal difference (TD) learning is a fundamental technique in reinforcement learning that updates value estimates for states or state-action pairs using a TD target. This target represents an improved estimate of the true value by incorporating both immediate rewards and the estimated value of subsequent states. Traditionally, TD learning relies on the value of a single subsequent state. We propose an enhanced multi-state TD (MSTD) target that utilizes the estimated values of multiple subsequent states. Building on this new MSTD concept, we develop complete actor-critic algorithms that include management of replay buffers in two modes, and integrate with deep deterministic policy optimization (DDPG) and soft actor-critic (SAC). Experimental results demonstrate that algorithms employing the MSTD target significantly improve learning performance compared to traditional methods.The code is provided on GitHub.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 6 of its 16 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

6 of 16 references.

References · 6

  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  2. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  3. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  4. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  5. 2020The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learningcited by 14
  6. 2018Fully Convolutional Network with Multi-Step Reinforcement Learning for Image Processingcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Parameter-Free Temporal Difference Learning
  2. 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion
  3. 2025Accelerating Multi-Task Temporal Difference Learning under Low-Rank Representation
  4. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning
  5. 2025Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits
  6. 2025Implicit Updates for Average-Reward Temporal Difference Learning
  7. 2025Analysis of Off-Policy n-Step TD-Learning with Linear Function Approximation
  8. 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning