mlsanityData to 14 Oct 2026

Zero-Shot Reinforcement Learning from Low Quality Data

arXiv 2309.15178 ↗First posted 26 Sep 2023cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Zero-shot reinforcement learning (RL) promises to provide agents that can perform any task in an environment after an offline, reward-free pre-training phase. Methods leveraging successor measures and successor features have shown strong performance in this setting, but require access to large heterogenous datasets for pre-training which cannot be expected for most real problems. Here, we explore how the performance of zero-shot RL methods degrades when trained on small homogeneous datasets, and propose fixes inspired by conservatism, a well-established feature of performant single-task offline RL algorithms. We evaluate our proposals across various datasets, domains and tasks, and show that conservative zero-shot RL algorithms outperform their non-conservative counterparts on low quality datasets, and perform no worse on high quality datasets. Somewhat surprisingly, our proposals also outperform baselines that get to see the task during training. Our code is available via https://enjeeneer.io/projects/zero-shot-rl/ .

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 76 of its 101 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

76 of 101 references.

References · 76

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
38 more
  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2018DeepMind Control Suitecited by 956
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2022A Generalist Agentcited by 808
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  12. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  13. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  14. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  15. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  16. 2019Challenges of Real-World Reinforcement Learningcited by 291
  17. 2019Self-Supervised Exploration via Disagreementcited by 265
  18. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  19. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  20. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  21. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  22. 2022Multi-Game Decision Transformerscited by 214
  23. 2022Online Decision Transformercited by 196
  24. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  25. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  26. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  27. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  28. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  29. 2020Model-Based Offline Planningcited by 129
  30. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  31. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  32. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  33. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  34. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  35. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  36. 2021Learning One Representation to Optimize All Rewardscited by 115
  37. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  38. 2021APS: Active Pretraining with Successor Featurescited by 112

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  2. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 22
  3. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  4. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  5. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  6. 2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
  7. 2025Epistemically-guided forward-backward explorationcited by 8
  8. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  9. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  10. 2025Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimizationcited by 4
  11. 2024Zero-Shot Offline Imitation Learning via Optimal Transportcited by 3
  12. 2026Zero-Shot Off-Policy Learningcited by 2
9 more
  1. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learningcited by 2
  2. 2025A Unified Framework for Zero-Shot Reinforcement Learningcited by 1
  3. 2026Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learningcited by 1
  4. 2026Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learningcited by 1
  5. 2026Improving Zero-Shot Offline RL via Behavioral Task Samplingcited by 1
  6. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 1
  7. 2026Evaluating Generalization Mechanisms in Autonomous Cyber Attack Agentscited by 0
  8. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  9. 2026When Dynamics Shift, Robust Task Inference Wins: Offline Imitation Learning with Behavior Foundation Models Revisitedcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On Zero-Shot Reinforcement Learning
  2. 2025Tackling the Zero-Shot Reinforcement Learning Loss Directly
  3. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
  4. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities
  5. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learning
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  7. 2024Distributional Successor Features Enable Zero-Shot Policy Optimization
  8. 2024Zero-Shot Reinforcement Learning via Function Encoders