Data to 14 Oct 2026

Simplifying Deep Temporal Difference Learning

arXiv 2407.04811 ↗First posted 5 Jul 2024cs.LGcited by 78

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Q-learning played a foundational role in the field reinforcement learning (RL). However, TD algorithms with off-policy data, such as Q-learning, or nonlinear function approximation like deep neural networks require several additional tricks to stabilise training, primarily a large replay buffer and target networks. Unfortunately, the delayed updating of frozen network parameters in the target network harms the sample efficiency and, similarly, the large replay buffer introduces memory and implementation overheads. In this paper, we investigate whether it is possible to accelerate and simplify off-policy TD training while maintaining its stability. Our key theoretical result demonstrates for the first time that regularisation techniques such as LayerNorm can yield provably convergent TD algorithms without the need for a target network or replay buffer, even with off-policy data. Empirically, we find that online, parallelised sampling enabled by vectorised environments stabilises training without the need for a large replay buffer. Motivated by these findings, we propose PQN, our simplified deep online Q-Learning algorithm. Surprisingly, this simple algorithm is competitive with more complex methods like: Rainbow in Atari, PPO-RNN in Craftax, QMix in Smax, and can be up to 50x faster than traditional DQN without sacrificing sample efficiency. In an era where PPO has become the go-to RL algorithm, PQN reestablishes off-policy Q-learning as a viable alternative.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 60 of its 94 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

60 of 94 references.

Closest ancestors · 2

Same direction, newest first

Other references · 58

2017Proximal Policy Optimization Algorithmscited by 15,215
2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
2016Layer Normalizationcited by 5,669
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2015Trust Region Policy Optimizationcited by 2,580
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786

38 more

2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
2017Counterfactual Multi-Agent Policy Gradientscited by 751
2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
2019The StarCraft Multi-Agent Challengecited by 597
2016Deep Exploration via Bootstrapped DQNcited by 527
2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
2018Distributed Prioritized Experience Replaycited by 269
2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
2019A Theoretical Analysis of Deep Q-Learningcited by 215
2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
2020The NetHack Learning Environmentcited by 198
2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
2018Deep Reinforcement Learning and the Deadly Triadcited by 174
2023Understanding plasticity in neural networkscited by 174
2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 166
2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
2019Behaviour Suite for Reinforcement Learningcited by 132
2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
2022Discovered Policy Optimisationcited by 104
2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
2021Off-Belief Learningcited by 75
2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
2017Frequentist Consistency of Variational Bayescited by 60
2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 58

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 38; 2026: 32.

78 papers.

Sort:

By area:

2025Hierarchical Reasoning Modelcited by 177
2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
2025Horizon Reduction Makes RL Scalablecited by 57
2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
2024Streaming Deep Reinforcement Learning Finally Workscited by 25
2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 20
2024NAVIX: Scaling MiniGrid Environments with JAXcited by 18
2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
2025Value-Based Deep RL Scales Predictablycited by 15

38 more

2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 13
2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
2025Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Taskscited by 11
2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
2025Hadamax Encoding: Elevating Performance in Model-Free Ataricited by 7
2025Relative Entropy Pathwise Policy Optimizationcited by 7
2025Replicable Reinforcement Learning with Linear Function Approximationcited by 7
2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
2025Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAXcited by 5
2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
2026SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Groundingcited by 5
2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
2025Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environmentscited by 4
2025Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Trainingcited by 4
2025Capability-Aware Shared Hypernetworks for Flexible Heterogeneous Multi-Robot Coordinationcited by 3
2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 3
2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
2025A Principle of Targeted Intervention for Multi-Agent Reinforcement Learningcited by 3
2025Understanding and Improving Hyperbolic Deep Reinforcement Learningcited by 3
2025Residual Policy Gradient: A Reward View of KL-regularized Objectivecited by 2
2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
2025Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Modelscited by 2
2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learningcited by 2
2025Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscalecited by 2
2026Temporal Difference Learning with Constrained Initial Representationscited by 2
2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
2026Benchmarking Open-Ended Multi-Agent Coordination in Language Agentscited by 2
2025JaxRobotarium: Training and Deploying Multi-Robot Policies in 10 Minutescited by 1
2025Universal Value-Function Uncertaintiescited by 1
2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theorycited by 1
2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
2026Periodic Regularized Q-Learningcited by 1
2026FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Controlcited by 1

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way