# Zero-Shot Reinforcement Learning from Low Quality Data

arXiv: https://arxiv.org/abs/2309.15178 · HTML: https://mlsanity.com/paper/2309.15178/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 101 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 30 of 76)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)

## Built on by (most cited first, 21 of 21)

- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.00739/) (2025)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weighting](https://mlsanity.com/paper/2412.04368/) (2024)
- [Intention-Conditioned Flow Occupancy Models](https://mlsanity.com/paper/2506.08902/) (2025)
- [Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics](https://mlsanity.com/paper/2505.13150/) (2025)
- [Epistemically-guided forward-backward exploration](https://mlsanity.com/paper/2507.05477/) (2025)
- [Towards Robust Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.15382/) (2025)
- [Zero-Shot Reinforcement Learning Under Partial Observability](https://mlsanity.com/paper/2506.15446/) (2025)
- [Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization](https://mlsanity.com/paper/2507.03051/) (2025)
- [Zero-Shot Offline Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2410.08751/) (2024)
- [Zero-Shot Off-Policy Learning](https://mlsanity.com/paper/2602.01962/) (2026)
- [A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2604.24532/) (2026)
- [A Unified Framework for Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.20542/) (2025)
- [Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2603.25464/) (2026)
- [Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2604.05931/) (2026)
- [Improving Zero-Shot Offline RL via Behavioral Task Sampling](https://mlsanity.com/paper/2604.25496/) (2026)
- [From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2606.01123/) (2026)
- [Evaluating Generalization Mechanisms in Autonomous Cyber Attack Agents](https://mlsanity.com/paper/2603.10041/) (2026)
- [Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning](https://mlsanity.com/paper/2605.13207/) (2026)
- [When Dynamics Shift, Robust Task Inference Wins: Offline Imitation Learning with Behavior Foundation Models Revisited](https://mlsanity.com/paper/2605.17017/) (2026)

## Abstract (arXiv)

Zero-shot reinforcement learning (RL) promises to provide agents that can perform any task in an environment after an offline, reward-free pre-training phase. Methods leveraging successor measures and successor features have shown strong performance in this setting, but require access to large heterogenous datasets for pre-training which cannot be expected for most real problems. Here, we explore how the performance of zero-shot RL methods degrades when trained on small homogeneous datasets, and propose fixes inspired by conservatism, a well-established feature of performant single-task offline RL algorithms. We evaluate our proposals across various datasets, domains and tasks, and show that conservative zero-shot RL algorithms outperform their non-conservative counterparts on low quality datasets, and perform no worse on high quality datasets. Somewhat surprisingly, our proposals also outperform baselines that get to see the task during training. Our code is available via https://enjeeneer.io/projects/zero-shot-rl/ .