mlsanityData to 7 Oct 2026

Mind the Model, Not the Agent: The Primacy Bias in Model-based RL

arXiv 2310.15017 ↗First posted 23 Oct 2023cs.LG cs.AIcited by 9

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

The primacy bias in model-free reinforcement learning (MFRL), which refers to the agent's tendency to overfit early data and lose the ability to learn from new data, can significantly decrease the performance of MFRL algorithms. Previous studies have shown that employing simple techniques, such as resetting the agent's parameters, can substantially alleviate the primacy bias in MFRL. However, the primacy bias in model-based reinforcement learning (MBRL) remains unexplored. In this work, we focus on investigating the primacy bias in MBRL. We begin by observing that resetting the agent's parameters harms its performance in the context of MBRL. We further find that the primacy bias in MBRL is more closely related to the primacy bias of the world model instead of the primacy bias of the agent. Based on this finding, we propose textitworld model resetting, a simple yet effective technique to alleviate the primacy bias in MBRL. We apply our method to two different MBRL algorithms, MBPO and DreamerV2. We validate the effectiveness of our method on multiple continuous control tasks on MuJoCo and DeepMind Control Suite, as well as discrete control tasks on Atari 100k benchmark. The experimental results show that textitworld model resetting can significantly alleviate the primacy bias in the model-based setting and improve the algorithm's performance. We also give a guide on how to perform textitworld model resetting effectively.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 38 of its 50 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

38 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting17 Mar
  2. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning16 Aug
  3. 2020Mastering Atari with Discrete World Models5 Oct
  4. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018DeepMind Control Suitecited by 935
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  8. 2019Model-Based Reinforcement Learning for Ataricited by 495
  9. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  10. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  12. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
21 more
  1. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  3. 2020Revisiting Fundamentals of Experience Replaycited by 171
  4. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  5. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  6. 2019When to use parametric models in reinforcement learning?cited by 118
  7. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  8. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  9. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  10. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  11. 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39
  12. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37
  13. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  14. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  15. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  16. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  17. 2023Curious Replay for Model-based Adaptationcited by 16
  18. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
  19. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  20. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
  21. 2018Mitigating Planner Overfitting in Model-Based Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 3.

9 papers.

  1. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  2. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  3. 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
  4. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
  5. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  6. 2024Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learningcited by 4
  7. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  8. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  9. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reset-free Reinforcement Learning with World Models
  2. 2022A Survey on Model-based Reinforcement Learning
  3. 2021Optimality Inductive Biases and Agnostic Guidelines for Offline Reinforcement Learning
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF
  7. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective