Data to 7 Oct 2026

Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability

arXiv 2107.06277 ↗First posted 13 Jul 2021cs.LG cs.AI stat.MLcited by 102

Not on the map. Why?

Abstract (arXiv)

Generalization is a central challenge for the deployment of reinforcement learning (RL) systems in the real world. In this paper, we show that the sequential structure of the RL problem necessitates new approaches to generalization beyond the well-studied techniques used in supervised learning. While supervised learning methods can generalize effectively without explicitly accounting for epistemic uncertainty, we show that, perhaps surprisingly, this is not the case in RL. We show that generalization to unseen test conditions from a limited number of training conditions induces implicit partial observability, effectively turning even fully-observed MDPs into POMDPs. Informed by this observation, we recast the problem of generalization in RL as solving the induced partially observed Markov decision process, which we call the epistemic POMDP. We demonstrate the failure modes of algorithms that do not appropriately handle this partial observability, and suggest a simple ensemble-based technique for approximately solving the partially observed problem. Empirically, we demonstrate that our simple algorithm derived from the epistemic POMDP achieves significant gains in generalization over current methods on the Procgen benchmark suite.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Builds on 41 of its 58 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

41 of 58 references.

References · 41

2017Proximal Policy Optimization Algorithmscited by 14,441
2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
2019Solving Rubik's Cube with a Robot Handcited by 699
2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
2016Deep Exploration via Bootstrapped DQNcited by 518
2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400

29 more

2018Quantifying Generalization in Reinforcement Learningcited by 345
2020Decoupling Representation Learning from Reinforcement Learningcited by 263
2013Learning to Optimize Via Posterior Samplingcited by 253
2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
2017Distral: Robust Multitask Reinforcement Learningcited by 231
2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
2020The NetHack Learning Environmentcited by 196
2020Prioritized Level Replaycited by 190
2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
2018Generalization and Regularization in DQNcited by 117
2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
2019Observational Overfitting in Reinforcement Learningcited by 90
2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
2017Divide-and-Conquer Reinforcement Learningcited by 46
2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
2019If MaxEnt RL is the Answer, What is the Question?cited by 27
2020Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learningcited by 26
2019Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flightcited by 25
2013Model-Based Bayesian Explorationcited by 21
2015Geometry and Determinism of Optimal Stationary Control in Partially Observable Markov Decision Processescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 17; 2023: 17; 2024: 23; 2025: 25; 2026: 17.

102 papers.

Sort:

By area:

2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
2022Evolving Curricula with Regret-Based Environment Designcited by 166
2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
2023Large Language Models for Robotics: A Surveycited by 132
2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
2022Offline RL Policies Should be Trained to be Adaptivecited by 52

38 more

2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 40
2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
2021Tell me why! Explanations support learning relational and causal structurecited by 36
2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
2025RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correctioncited by 35
2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 28
2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
2023Explore to Generalize in Zero-Shot RLcited by 20
2023Learning to Extrapolate: A Transductive Approachcited by 19
2024Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptationcited by 19
2022Generalization in Cooperative Multi-Agent Systemscited by 18
2024Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-Making Frameworkcited by 16
2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
2025CoLMDriver: LLM-based Negotiation Benefits Cooperative Autonomous Drivingcited by 13
2022Generalized Data Distribution Iterationcited by 12
2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
2024Entity-based Reinforcement Learning for Autonomous Cyber Defencecited by 11
2022The Impact of Task Underspecification in Evaluating Deep Reinforcement Learningcited by 10
2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learningcited by 9
2024RELIEF: Reinforcement Learning Empowered Graph Feature Prompt Tuningcited by 9
2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
2022On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardnesscited by 7
2024SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systemscited by 7
2024Certifiably Robust Policies for Uncertain Parametric Environmentscited by 6
2024IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learningcited by 6
2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
2023Task Aware Dreamer for Task Generalization in Reinforcement Learningcited by 5
2023SAGE: Bridging Semantic and Actionable Parts for GEneralizable Manipulation of Articulated Objectscited by 5
2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
2022Learn-to-Race Challenge 2022: Benchmarking Safe Learning and Cross-domain Generalisation in Autonomous Racingcited by 4
2025Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lenscited by 4
2025Imagined Autocurriculacited by 4
2025BuilderBench: The Building Blocks of Intelligent Agentscited by 4

50 most cited of 102 All 102 (JSON)