mlsanityData to 14 Oct 2026

Observational Overfitting in Reinforcement Learning

arXiv 1912.02975 ↗First posted 6 Dec 2019cs.LG cs.AI stat.MLcited by 91

Not on the map. Why?

Cited by 25 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 23; 2023: 21; 2024: 11; 2025: 17; 2026: 3.

91 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  3. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  4. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  5. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  6. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  7. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  8. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  9. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  10. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  11. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  12. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
38 more
  1. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  2. 2021Adversarially Guided Actor-Criticcited by 47
  3. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 43
  4. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 42
  5. 2023Goodhart's Law in Reinforcement Learningcited by 41
  6. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  7. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  8. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  9. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  10. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  11. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  12. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  13. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  14. 2022Hyperbolic Deep Reinforcement Learningcited by 25
  15. 2024MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learningcited by 23
  16. 2021When Is Generalizable Reinforcement Learning Tractable?cited by 20
  17. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 20
  18. 2022Online Policy Optimization for Robust MDPcited by 19
  19. 2023Gradient-based Planning with World Modelscited by 19
  20. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  21. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  22. 2022Adversarial Cheap Talkcited by 16
  23. 2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15
  24. 2022Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributionscited by 15
  25. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 14
  26. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 14
  27. 2021Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixturecited by 12
  28. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  29. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  30. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  31. 2022Local Feature Swapping for Generalization in Reinforcement Learningcited by 9
  32. 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
  33. 2022Rethinking Value Function Learning for Generalization in Reinforcement Learningcited by 8
  34. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 7
  35. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 6
  36. 2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scalecited by 6
  37. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
  38. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learningcited by 5

50 most cited of 91 All 91 (JSON)