mlsanityData to 7 Oct 2026

The Phenomenon of Policy Churn

arXiv 2206.00730 ↗First posted 1 Jun 2022cs.LG cs.AI stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

We identify and study the phenomenon of policy churn, that is, the rapid change of the greedy policy in value-based reinforcement learning. Policy churn operates at a surprisingly rapid pace, changing the greedy action in a large fraction of states within a handful of learning updates (in a typical deep RL set-up such as DQN on Atari). We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties. A number of ablations help whittle down the plausible explanations on why churn occurs to just a handful, all related to deep learning. Finally, we hypothesise that policy churn is a beneficial but overlooked form of implicit exploration that casts ε-greedy exploration in a fresh light, namely that ε-noise plays a much smaller role than expected.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 23 of its 57 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

23 of 57 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2016An overview of gradient descent optimization algorithmscited by 1,021
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
11 more
  1. 2017Noisy Networks for Explorationcited by 262
  2. 2019Behaviour Suite for Reinforcement Learningcited by 129
  3. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  4. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  5. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  6. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  7. 2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 22
  8. 2021When should agents explore?cited by 20
  9. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  10. 2019Adapting Behaviour for Learning Progresscited by 11
  11. 2021Estimates for the Branching Factors of Atari Gamescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 8; 2025: 8; 2026: 2.

28 papers.

Sort:

By area:

  1. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  2. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  3. 2022Human-level Atari 200x fastercited by 28
  4. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  5. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  6. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  7. 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 16
  8. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  9. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  10. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  11. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  12. 2023Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Controlcited by 11
16 more
  1. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  2. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  3. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  4. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  5. 2025Replicable Reinforcement Learning with Linear Function Approximationcited by 6
  6. 2023Measuring and Mitigating Interference in Reinforcement Learningcited by 5
  7. 2023An Invitation to Deep Reinforcement Learningcited by 5
  8. 2024CALE: Continuous Arcade Learning Environmentcited by 5
  9. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  10. 2023The Ladder in Chaos: A Simple and Effective Improvement to General DRL Algorithms by Policy Path Trimming and Boostingcited by 1
  11. 2024Issues with Value-Based Multi-objective Reinforcement Learning: Value Function Interference and Overestimation Sensitivitycited by 1
  12. 2025Deep Double Q-learningcited by 1
  13. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  14. 2025ES-C51: Expected Sarsa Based C51 Distributional Reinforcement Learning Algorithmcited by 0
  15. 2025Forgetting is Everywherecited by 0
  16. 2026Scalable Reinforcement Learning via Adaptive Batch Scalingcited by 0