mlsanityData to 7 Oct 2026

Explore to Generalize in Zero-Shot RL

arXiv 2306.03072 ↗First posted 5 Jun 2023cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

We study zero-shot generalization in reinforcement learning-optimizing a policy on a set of training tasks to perform well on a similar but unseen test task. To mitigate overfitting, previous work explored different notions of invariance to the task. However, on problems such as the ProcGen Maze, an adequate solution that is invariant to the task visualization does not exist, and therefore invariance-based approaches fail. Our insight is that learning a policy that effectively textitexplores the domain is harder to memorize than a policy that maximizes reward for a specific task, and therefore we expect such learned behavior to generalize well; we indeed demonstrate this empirically on several domains that are difficult for invariance-based approaches. Our textitExplore to Generalize algorithm (ExpGen) builds on this insight: we train an additional ensemble of agents that optimize reward. At test time, either the ensemble agrees on an action, and we generalize well, or we take exploratory actions, which generalize well and drive us to a novel part of the state space, where the ensemble may potentially agree again. We show that our approach is the state-of-the-art on tasks of the ProcGen challenge that have thus far eluded effective generalization, yielding a success rate of 83% on the Maze task and 74% on Heist with 200 training levels. ExpGen can also be combined with an invariance based approach to gain the best of both worlds, setting new state-of-the-art results on ProcGen.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

38 of 56 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  8. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  9. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  10. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  11. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  12. 2020Prioritized Level Replaycited by 190
26 more
  1. 2019Efficient Exploration via State Marginal Matchingcited by 186
  2. 2021Reinforcement Learning with Prototypical Representationscited by 179
  3. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  4. 2016Value Iteration Networkscited by 158
  5. 2020Phasic Policy Gradientcited by 121
  6. 2018Generalization and Regularization in DQNcited by 117
  7. 2021APS: Active Pretraining with Successor Featurescited by 109
  8. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  9. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  10. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  11. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  12. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  13. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  14. 2020Deep Reinforcement and InfoMax Learningcited by 79
  15. 2017Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physicscited by 57
  16. 2021Meta-Learning with Fewer Tasks through Task Interpolationcited by 45
  17. 2020Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learningcited by 37
  18. 2020Generalized Planning With Deep Reinforcement Learningcited by 35
  19. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  20. 2017Action Schema Networks: Generalised Policies with Deep Learningcited by 32
  21. 2020Rotation, Translation, and Cropping for Zero-Shot Generalizationcited by 23
  22. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  23. 2021Domain Adversarial Reinforcement Learningcited by 19
  24. 2020Instance based Generalization in Reinforcement Learningcited by 17
  25. 2021Neuro-algorithmic Policies enable Fast Combinatorial Generalizationcited by 11
  26. 2021Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stabilitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 6; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025Episodic Novelty Through Temporal Distancecited by 15
  2. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  3. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  4. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  5. 2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scalecited by 6
  6. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  7. 2024Test-Time Regret Minimization in Meta Reinforcement Learningcited by 5
  8. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  9. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  10. 2024Inductive Generalization in Reinforcement Learning from Specificationscited by 3
  11. 2025Filtering Learning Histories Enhances In-Context Reinforcement Learningcited by 3
  12. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
8 more
  1. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 1
  2. 2025Blindfolded Experts Generalize Better: Insights from Robotic Manipulation and Videogamescited by 1
  3. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0
  4. 2026KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learningcited by 0
  5. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  6. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0
  7. 2026Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learningcited by 0
  8. 2026Robust Successor Featurescited by 0