mlsanityData to 7 Oct 2026

MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL

arXiv 2410.08896 ↗First posted 11 Oct 2024cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Building deep reinforcement learning (RL) agents that find a good policy with few samples has proven notoriously challenging. To achieve sample efficiency, recent work has explored updating neural networks with large numbers of gradient steps for every new sample. While such high update-to-data (UTD) ratios have shown strong empirical performance, they also introduce instability to the training process. Previous approaches need to rely on periodic neural network parameter resets to address this instability, but restarting the training process is infeasible in many real-world applications and requires tuning the resetting interval. In this paper, we focus on one of the core difficulties of stable training with limited samples: the inability of learned value functions to generalize to unobserved on-policy actions. We mitigate this issue directly by augmenting the off-policy RL training process with a small amount of data generated from a learned world model. Our method, Model-Augmented Data for TD Learning (MAD-TD), uses small amounts of generated data to stabilize high UTD training and achieve competitive performance on the most challenging tasks in the DeepMind control suite. Our experiments further highlight the importance of employing a good model to generate data, MAD-TD's ability to combat value overestimation, and its practical stability gains for continued learning.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 74 of its 98 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

74 of 98 references.

References · 74

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
38 more
  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  4. 2017Robust Adversarial Reinforcement Learningcited by 439
  5. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  6. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  7. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  8. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  9. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  11. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  12. 2020Revisiting Fundamentals of Experience Replaycited by 171
  13. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  14. 2023Understanding plasticity in neural networkscited by 170
  15. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  16. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  17. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  18. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  19. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  20. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
  21. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  22. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  23. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  24. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  25. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  26. 2023Synthetic Experience Replaycited by 100
  27. 2016The Predictron: End-To-End Learning and Planningcited by 92
  28. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  29. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  30. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  31. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  32. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  33. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  34. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  35. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  36. 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
  37. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  38. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  2. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
  3. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  4. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  5. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  6. 2025Replicable Reinforcement Learning with Linear Function Approximationcited by 6
  7. 2025Calibrated Value-Aware Model Learning with Probabilistic Environment Modelscited by 2
  8. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  9. 2026Reward-Conditioned Reinforcement Learningcited by 1
  10. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  11. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  12. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
4 more
  1. 2025Iterative Compositional Data Generation for Robot Controlcited by 0
  2. 2026R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learningcited by 0
  3. 2026Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitionerscited by 0
  4. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  2. 2023Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting
  3. 2023HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approach
  4. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  5. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
  6. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  7. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
  8. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning