mlsanityData to 14 Oct 2026

Provable Zero-Shot Generalization in Offline Reinforcement Learning

arXiv 2503.07988 ↗First posted 11 Mar 2025cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 64 of its 74 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

64 of 74 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024Offline Multi-task Transfer RL with Representational Penalization19 Feb
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond6 Jan
  3. 2023The Generalization Gap in Offline Reinforcement Learning10 Dec
  4. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  5. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  7. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  8. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions29 Nov
  9. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  10. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  11. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  12. 2021Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RL22 Jun
9 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  2. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  6. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2016Deep Exploration via Bootstrapped DQNcited by 527
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  6. 2017Robust Adversarial Reinforcement Learningcited by 451
  7. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  8. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  9. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  10. 2020The NetHack Learning Environmentcited by 198
  11. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  12. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
31 more
  1. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  2. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  3. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  4. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  5. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  6. 2024Foundation Policies with Hilbert Representationscited by 94
  7. 2019Observational Overfitting in Reinforcement Learningcited by 91
  8. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  9. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  10. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 81
  11. 2017Towards Generalization and Simplicity in Continuous Controlcited by 68
  12. 2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
  13. 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
  14. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  15. 2021Near-optimal Representation Learning for Linear Bandits and Linear RLcited by 45
  16. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  17. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 42
  18. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  19. 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
  20. 2020Rotation, Translation, and Cropping for Zero-Shot Generalizationcited by 24
  21. 2019On the Generalization Gap in Reparameterizable Reinforcement Learningcited by 22
  22. 2021When Is Generalizable Reinforcement Learning Tractable?cited by 20
  23. 2020Instance based Generalization in Reinforcement Learningcited by 17
  24. 2022Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worldscited by 17
  25. 2021Provably Efficient Multi-Task Reinforcement Learning with Model Transfercited by 16
  26. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  27. 2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Boundscited by 11
  28. 2020Sequential Transfer in Reinforcement Learning with a Generative Modelcited by 10
  29. 2022On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardnesscited by 7
  30. 2022Powderworld: A Platform for Understanding Generalization via Rich Task Distributionscited by 6
  31. 2025Towards Understanding the Benefit of Multitask Representation Learning in Decision Processcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluationcited by 0
  2. 2026Formalizing Task-Space Complexity for Zero-Shot Generalizationcited by 0
  3. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learning
  2. 2026Improving Zero-Shot Offline RL via Behavioral Task Sampling
  3. 2020POPO: Pessimistic Offline Policy Optimization
  4. 2023Zero-Shot Reinforcement Learning from Low Quality Data
  5. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  6. 2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning
  7. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
  8. 2023Model-based Offline Reinforcement Learning with Local Misspecification