mlsanityData to 14 Oct 2026

Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RL

arXiv 2306.02419 ↗First posted 4 Jun 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning agents tend to develop habits that are effective only under specific policies. Following an initial exploration phase where agents try out different actions, they eventually converge onto a particular policy. As this occurs, the distribution over state-action trajectories becomes narrower, leading agents to repeatedly experience the same transitions. This repetitive exposure fosters spurious correlations between certain observations and rewards. Agents may then pick up on these correlations and develop simplistic habits tailored to the specific set of trajectories dictated by their policy. The problem is that these habits may yield incorrect outcomes when agents are forced to deviate from their typical trajectories, prompted by changes in the environment. This paper presents a mathematical characterization of this phenomenon, termed policy confounding, and illustrates, through a series of examples, the circumstances under which it occurs.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 26 of its 59 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

26 of 59 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  5. 2018Recognition in Terra Incognitacited by 658
  6. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  7. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  8. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  9. 2018An Introduction to Deep Reinforcement Learningcited by 206
  10. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  11. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  12. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
14 more
  1. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  2. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  3. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  4. 2020Invariant Causal Prediction for Block MDPscited by 105
  5. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  6. 2019Observational Overfitting in Reinforcement Learningcited by 91
  7. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  8. 2021Out of Distribution Generalization in Machine Learningcited by 60
  9. 2020Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learningcited by 53
  10. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  11. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  12. 2021Invariant Policy Learning: A Causal Perspectivecited by 17
  13. 2023Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectoriescited by 9
  14. 2021Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 2.

10 papers.

  1. 2025Better Decisions through the Right Causal World Modelcited by 3
  2. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  3. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  4. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learningcited by 1
  5. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  6. 2025Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applicationscited by 1
  7. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0
  8. 2025Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representationscited by 0
  9. 2026Dynamic Context Scheduling: Learning Beyond the Static Universecited by 0
  10. 2026Reinforcement Learning for Symbolic Equation Solvingcited by 0