mlsanityData to 7 Oct 2026

Does Zero-Shot Reinforcement Learning Exist?

arXiv 2209.14935 ↗First posted 29 Sep 2022cs.LGcited by 79

Not on the map. Why?

Abstract (arXiv)

A zero-shot RL agent is an agent that can solve any RL task in a given environment, instantly with no additional planning or learning, after an initial reward-free learning phase. This marks a shift from the reward-centric RL paradigm towards "controllable" agents that can follow arbitrary instructions in an environment. Current RL agents can solve families of related tasks at best, or require planning anew for each task. Strategies for approximate zero-shot RL ave been suggested using successor features (SFs) [BBQ+ 18] or forward-backward (FB) representations [TO21], but testing has been limited. After clarifying the relationships between these schemes, we introduce improved losses and new SF models, and test the viability of zero-shot RL schemes systematically on tasks from the Unsupervised RL benchmark [LYL+21]. To disentangle universal representation learning from exploration, we work in an offline setting and repeat the tests on several existing replay buffers. SFs appear to suffer from the choice of the elementary state features. SFs with Laplacian eigenfunctions do well, while SFs based on auto-encoders, inverse curiosity, transition models, low-rank transition matrix, contrastive learning, or diversity (APS), perform unconsistently. In contrast, FB representations jointly learn the elementary and successor features from a single, principled criterion. They perform best and consistently across the board, reaching 85% of supervised RL performance with a good replay buffer, in a zero-shot manner.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

31 of 56 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2018DeepMind Control Suitecited by 935
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  12. 2016Variational Intrinsic Controlcited by 262
19 more
  1. 2021Reinforcement Learning with Prototypical Representationscited by 179
  2. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  3. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  4. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  5. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  6. 2021Learning One Representation to Optimize All Rewardscited by 110
  7. 2021APS: Active Pretraining with Successor Featurescited by 109
  8. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  9. 2018Universal Successor Features Approximatorscited by 102
  10. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  11. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  12. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 67
  13. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  14. 2016Deep Reinforcement Learning with Successor Features for Navigation across Similar Environmentscited by 50
  15. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  16. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  17. 2020Universal Successor Features for Transfer Reinforcement Learningcited by 20
  18. 2019Disentangled Cumulants Help Successor Representations Transfer to New Taskscited by 11
  19. 2020Zero-Shot Reinforcement Learning with Deep Attention Convolutional Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 21; 2025: 25; 2026: 23.

79 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2025Reinforcement Learning with Action Chunkingcited by 97
  3. 2024Foundation Policies with Hilbert Representationscited by 89
  4. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  5. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  6. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  7. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  8. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  9. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28
  10. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  11. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  12. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
38 more
  1. 2025Temporal Difference Flowscited by 20
  2. 2025Fast Adaptation with Behavioral Foundation Modelscited by 20
  3. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 17
  4. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  5. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 17
  6. 2024A Distributional Analogue to the Successor Representationcited by 15
  7. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  8. 2025Dual Goal Representationscited by 13
  9. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  10. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  11. 2024Learning Successor Features the Simple Waycited by 10
  12. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  13. 2023Proper Laplacian Representation Learningcited by 9
  14. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  15. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  16. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  17. 2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
  18. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 7
  19. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  20. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  21. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  22. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  23. 2025Epistemically-guided forward-backward explorationcited by 6
  24. 2025Optimistic Task Inference for Behavior Foundation Modelscited by 5
  25. 2024Predictive representations: building blocks of intelligencecited by 4
  26. 2024Few-Shot Task Learning through Inverse Generative Modelingcited by 4
  27. 2026Compositional Planning with Jumpy World Modelscited by 4
  28. 2023A State Representation for Diminishing Rewardscited by 3
  29. 2023Goal-conditioned Offline Planning from Curious Explorationcited by 3
  30. 2024Zero-Shot Offline Imitation Learning via Optimal Transportcited by 3
  31. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
  32. 2025Reward-Aware Proto-Representations in Reinforcement Learningcited by 3
  33. 2025Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Explorationcited by 3
  34. 2025Constructing an Optimal Behavior Basis for the Option Keyboardcited by 2
  35. 2025Goal-Driven Reward by Video Diffusion Models for Reinforcement Learningcited by 2
  36. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 2
  37. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learningcited by 2
  38. 2023Learning Successor Features with Distributed Hebbian Temporal Memorycited by 1

50 most cited of 79 All 79 (JSON)