mlsanityData to 14 Oct 2026

The Primacy Bias in Deep Reinforcement Learning

arXiv 2205.07802 ↗First posted 16 May 2022cs.LG cs.AI stat.MLcited by 247

Not on the map. Why?

Cited by 128 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.

Key facts. Cited by 247 papers in the map (data through 14 Oct 2026). Builds on 35 of its 72 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

35 of 72 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  11. 2019Model-Based Reinforcement Learning for Ataricited by 502
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
23 more
  1. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  2. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  3. 2017Distral: Robust Multitask Reinforcement Learningcited by 233
  4. 2020Revisiting Fundamentals of Experience Replaycited by 175
  5. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  6. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  7. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  8. 2019When to use parametric models in reinforcement learning?cited by 118
  9. 2019Are All Layers Created Equal?cited by 94
  10. 2019Observational Overfitting in Reinforcement Learningcited by 91
  11. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  12. 2020Compositional Languages Emerge in a Neural Iterated Learning Modelcited by 75
  13. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  14. 2018What Doubling Tricks Can and Can't Do for Multi-Armed Banditscited by 71
  15. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  16. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  17. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  18. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
  19. 2021The Impact of Reinitialization on Generalization in Convolutional Neural Networkscited by 20
  20. 2021Knowledge Evolution in Neural Networkscited by 18
  21. 2021On the Convergence of Stochastic Extragradient for Bilinear Games using Restarted Iteration Averagingcited by 18
  22. 2020How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimizationcited by 16
  23. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithmscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 48; 2024: 66; 2025: 58; 2026: 70.

247 papers.

Sort:

By area:

  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  2. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  3. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  4. 2023Understanding plasticity in neural networkscited by 174
  5. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  6. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  7. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  8. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  9. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  10. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  11. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  12. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
38 more
  1. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  2. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  3. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  4. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 63
  5. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  6. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  7. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 58
  8. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  9. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  10. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  11. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  12. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  13. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  14. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  15. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  16. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  17. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  18. 2023Small batch deep reinforcement learningcited by 36
  19. 2023Language Model Alignment with Elastic Resetcited by 34
  20. 2024Learning Continually by Spectral Regularizationcited by 34
  21. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  22. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 30
  23. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  24. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 28
  25. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  26. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  27. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  28. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  29. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 26
  30. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  31. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 24
  32. 2024MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learningcited by 23
  33. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  34. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  35. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  36. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  37. 2023Prediction and Control in Continual Reinforcement Learningcited by 19
  38. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19

50 most cited of 247 All 247 (JSON)