# Measuring and Characterizing Generalization in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1812.02868 · HTML: https://mlsanity.com/paper/1812.02868/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2018.

## Built on by (most cited first, 20 of 20)

- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Procedural Generalization by Planning with Self-Supervised World Models](https://mlsanity.com/paper/2111.01587/) (2021)
- [Stabilizing Unsupervised Environment Design with a Learned Adversary](https://mlsanity.com/paper/2308.10797/) (2023)
- [Language and Culture Internalisation for Human-Like Autotelic AI](https://mlsanity.com/paper/2206.01134/) (2022)
- [Correcting Momentum in Temporal Difference Learning](https://mlsanity.com/paper/2106.03955/) (2021)
- [Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shaping](https://mlsanity.com/paper/2209.14271/) (2022)
- [Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex](https://mlsanity.com/paper/2211.14673/) (2022)
- [Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions](https://mlsanity.com/paper/2310.03195/) (2023)
- [Explainable Artificial Intelligence (XAI) for Increasing User Trust in Deep Reinforcement Learning Driven Autonomous Systems](https://mlsanity.com/paper/2106.03775/) (2021)
- [Explaining Reinforcement Learning Policies through Counterfactual Trajectories](https://mlsanity.com/paper/2201.12462/) (2022)
- [Learning Generalizable Behavior via Visual Rewrite Rules](https://mlsanity.com/paper/2112.05218/) (2021)
- [Does DQN really learn? Exploring adversarial training schemes in Pong](https://mlsanity.com/paper/2203.10614/) (2022)
- [Measuring Interventional Robustness in Reinforcement Learning](https://mlsanity.com/paper/2209.09058/) (2022)
- [Reward-Predictive Clustering](https://mlsanity.com/paper/2211.03281/) (2022)
- [Policy-Induced Self-Supervision Improves Representation Finetuning in Visual RL](https://mlsanity.com/paper/2302.06009/) (2023)
- [Novelty Accommodating Multi-Agent Planning in High Fidelity Simulated Open World](https://mlsanity.com/paper/2306.12654/) (2023)
- [A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networks](https://mlsanity.com/paper/2307.07575/) (2023)
- [Rapid Open-World Adaptation by Adaptation Principles Learning](https://mlsanity.com/paper/2312.11138/) (2023)
- [An Efficient Approach for Cooperative Multi-Agent Learning Problems](https://mlsanity.com/paper/2504.04850/) (2025)
- [Adaptive Control in Autonomous Driving via Real-Time Recurrent RL](https://mlsanity.com/paper/2602.02236/) (2026)

## Abstract (arXiv)

Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-policy, and unreachable states. We propose a set of practical methods for evaluating agents with these definitions of generalization. We demonstrate these techniques on a common benchmark task for deep RL, and we show that the learned networks make poor decisions for states that differ only slightly from on-policy states, even though those states are not selected adversarially. Taken together, these results call into question the extent to which deep Q-networks learn generalized representations, and suggest that more experimentation and analysis is necessary before claims of representation learning can be supported.