mlsanityData to 14 Oct 2026

Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting

arXiv 2303.10144 ↗First posted 17 Mar 2023cs.LG stat.MLcited by 4

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Early stopping based on the validation set performance is a popular approach to find the right balance between under- and overfitting in the context of supervised learning. However, in reinforcement learning, even for supervised sub-problems such as world model learning, early stopping is not applicable as the dataset is continually evolving. As a solution, we propose a new general method that dynamically adjusts the update to data (UTD) ratio during training based on under- and overfitting detection on a small subset of the continuously collected experience not used for training. We apply our method to DreamerV2, a state-of-the-art model-based reinforcement learning algorithm, and evaluate it on the DeepMind Control Suite and the Atari 100k benchmark. The results demonstrate that one can better balance under- and overestimation by adjusting the UTD ratio with our approach compared to the default setting in DreamerV2 and that it is competitive with an extensive hyperparameter search which is not feasible for many applications. Our method eliminates the need to set the UTD hyperparameter by hand and even leads to a higher robustness with regard to other learning-related hyperparameters further reducing the amount of necessary tuning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 23 of its 35 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

23 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May

Other references · 20

  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  4. 2018DeepMind Control Suitecited by 956
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2019Model-Based Reinforcement Learning for Ataricited by 502
  7. 2020Reinforcement Learning with Augmented Datacited by 468
  8. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  9. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  10. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  11. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  12. 2019Observational Overfitting in Reinforcement Learningcited by 91
8 more
  1. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  2. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  3. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  4. 2018Bayesian Optimization in AlphaGocited by 55
  5. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42
  6. 2019Adapting Behaviour for Learning Progresscited by 11
  7. 2018Mitigating Planner Overfitting in Model-Based Reinforcement Learningcited by 9
  8. 2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 2.

4 papers.

  1. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  2. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  3. 2026Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptationcited by 1
  4. 2026Optimal Training-Time Scaling in Gradual Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
  2. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning
  3. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergence
  4. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
  5. 2025Compute-Optimal Scaling for Value-Based Deep RL
  6. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  7. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  8. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization