mlsanityData to 7 Oct 2026

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

arXiv 2111.09794 ↗First posted 18 Nov 2021cs.LG cs.AIcited by 164

Not on the map. Why?

Abstract (arXiv)

The study of zero-shot generalisation (ZSG) in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We rely on a unifying formalism and terminology for discussing different ZSG problems, building upon previous works. We go on to categorise existing benchmarks for ZSG, as well as current methods for tackling these problems. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in ZSG, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for ZSG, and we recommend building benchmarks in underexplored problem settings such as offline RL ZSG and reward-function variation.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Builds on 170 of its 209 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

170 of 209 references.

References · 170

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2019Invariant Risk Minimizationcited by 1,832
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  11. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  12. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
38 more
  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  3. 2018DeepMind Control Suitecited by 935
  4. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  5. 2021Domain Generalization: A Surveycited by 760
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2017Deep Reinforcement Learning that Matterscited by 727
  8. 2019Solving Rubik's Cube with a Robot Handcited by 699
  9. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  13. 2021Domain Generalization with MixStylecited by 563
  14. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  15. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  16. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  17. 2017Robust Adversarial Reinforcement Learningcited by 439
  18. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  19. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  20. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  21. 2016Learning to reinforcement learncited by 369
  22. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  23. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  24. 2018Maximum a Posteriori Policy Optimisationcited by 318
  25. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  26. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  27. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  28. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  29. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  30. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  31. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  32. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  33. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  34. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  35. 2019Causality for Machine Learningcited by 242
  36. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  37. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  38. 2015Contextual Markov Decision Processescited by 213

50 most cited of 170 All 170 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 27; 2024: 49; 2025: 43; 2026: 38.

164 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  3. 2024Robust agents learn causal world modelscited by 81
  4. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  5. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  6. 2024Policy-Guided Diffusioncited by 40
  7. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  8. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  9. 2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 31
  10. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  11. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  12. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
38 more
  1. 2024Machine Learning Augmented Branch and Bound for Mixed Integer Linear Programmingcited by 24
  2. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  3. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 23
  4. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
  5. 2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
  6. 2024Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-Making Frameworkcited by 16
  7. 2023Games for Artificial Intelligence Research: A Review and Perspectivescited by 14
  8. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  9. 2023Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentationcited by 13
  10. 2024Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Searchcited by 13
  11. 2025CoLMDriver: LLM-based Negotiation Benefits Cooperative Autonomous Drivingcited by 13
  12. 2024A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learningcited by 12
  13. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  14. 2024Entity-based Reinforcement Learning for Autonomous Cyber Defencecited by 11
  15. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  16. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  17. 2022Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shapingcited by 8
  18. 2022On Realization of Intelligent Decision-Making in the Real World: A Foundation Decision Model Perspectivecited by 8
  19. 2023AutoRL Hyperparameter Landscapescited by 8
  20. 2023Maximum diffusion reinforcement learningcited by 8
  21. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learningcited by 8
  22. 2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
  23. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  24. 2024Structural Generalization in Autonomous Cyber Incident Response with Message-Passing Neural Networks and Reinforcement Learningcited by 7
  25. 2024Model-Based Transfer Learning for Contextual Reinforcement Learningcited by 7
  26. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  27. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  28. 2022Powderworld: A Platform for Understanding Generalization via Rich Task Distributionscited by 6
  29. 2023Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learningcited by 6
  30. 2024Hierarchical Transformers are Efficient Meta-Reinforcement Learnerscited by 6
  31. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
  32. 2025Beyond Statistical Learning: Exact Learning Is Essential for General Intelligencecited by 6
  33. 2025Automated Cyber Defense with Generalizable Graph-based Reinforcement Learning Agentscited by 6
  34. 2026Generalization in Online Reinforcement Learning for Mobile Agentscited by 6
  35. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  36. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  37. 2024Instance Selection for Dynamic Algorithm Configuration with Reinforcement Learning: Improving Generalizationcited by 5
  38. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5

50 most cited of 164 All 164 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Zero-Shot Reinforcement Learning from Low Quality Data
  2. 2026Improving Zero-Shot Offline RL via Behavioral Task Sampling
  3. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions
  4. 2024Distributional Successor Features Enable Zero-Shot Policy Optimization
  5. 2025Towards Robust Zero-Shot Reinforcement Learning
  6. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspective