mlsanityData to 14 Oct 2026

On the Convergence of Semi-Gradient TD: A Mean Field Perspective

arXiv 2610.20014 ↗First posted 14 Oct 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability. We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target. Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution. We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 10 of its 15 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

10 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks7 May

Other references · 9

  1. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 61
  2. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 55
  3. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 50
  4. 2023Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reductioncited by 20
  5. 2023TD Convergence: An Optimization Perspectivecited by 16
  6. 2019Geometric Insights into the Convergence of Nonlinear TD Learningcited by 13
  7. 2021Sample Complexity and Overparameterization Bounds for Temporal Difference Learning with Neural Network Approximationcited by 10
  8. 2020Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theorycited by 8
  9. 2019Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  2. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation
  3. 2026Behavior-Consistent Deep Reinforcement Learning
  4. 2026Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
  5. 2025First-order Sobolev Reinforcement Learning
  6. 2026Entropy-Preserving Reinforcement Learning
  7. 2026Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning
  8. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State