mlsanityData to 7 Oct 2026

Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings

arXiv 2402.17135 ↗First posted 27 Feb 2024cs.LG cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Can we pre-train a generalist agent from a large amount of unlabeled offline trajectories such that it can be immediately adapted to any new downstream tasks in a zero-shot manner? In this work, we present a functional reward encoding (FRE) as a general, scalable solution to this zero-shot RL problem. Our main idea is to learn functional representations of any arbitrary tasks by encoding their state-reward samples using a transformer-based variational auto-encoder. This functional encoding not only enables the pre-training of an agent from a wide diversity of general unsupervised reward functions, but also provides a way to solve any new downstream tasks in a zero-shot manner, given a small number of reward-annotated samples. We empirically show that FRE agents trained on diverse random unsupervised reward functions can generalize to solve novel tasks in a range of simulated robotic benchmarks, often outperforming previous zero-shot RL and offline RL methods. Code for this project is provided at: https://github.com/kvfrans/fre

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

45 of 59 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Segment Anythingcited by 8,158
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2018DeepMind Control Suitecited by 935
  9. 2016Deep Variational Information Bottleneckcited by 915
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
33 more
  1. 2018Conditional Neural Processescited by 421
  2. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  4. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  5. 2018Neural Processescited by 307
  6. 2019Attentive Neural Processescited by 280
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  8. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  9. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  10. 2015Contextual Markov Decision Processescited by 213
  11. 2018Semi-parametric Topological Memory for Navigationcited by 203
  12. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  13. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  14. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  15. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  16. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  17. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  18. 2021Learning One Representation to Optimize All Rewardscited by 110
  19. 2018Universal Successor Features Approximatorscited by 102
  20. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  21. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  22. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  23. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
  24. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  25. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  26. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  27. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  28. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  29. 2021Learning more skills through optimistic explorationcited by 37
  30. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 33
  31. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  32. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  33. 2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 10.

31 papers.

Sort:

By area:

  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2025Reinforcement Learning with Action Chunkingcited by 97
  3. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  4. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  5. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  6. 2025A Survey of Continual Reinforcement Learningcited by 16
  7. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  8. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  9. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  10. 2025Value Flowscited by 12
  11. 2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
  12. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
19 more
  1. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  2. 2026Compositional Planning with Jumpy World Modelscited by 4
  3. 2024Zero-Shot Offline Imitation Learning via Optimal Transportcited by 3
  4. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2
  5. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 2
  6. 2024LoopSR: Looping Sim-and-Real for Lifelong Policy Adaptation of Legged Robotscited by 1
  7. 2025What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representationscited by 1
  8. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  9. 2026Reward-Conditioned Reinforcement Learningcited by 1
  10. 2026Improving Zero-Shot Offline RL via Behavioral Task Samplingcited by 1
  11. 2026Exploration and Online Transfer with Behavioral Foundation Modelscited by 1
  12. 2024One-shot World Models Using a Transformer Trained on a Synthetic Priorcited by 0
  13. 2025Tackling the Zero-Shot Reinforcement Learning Loss Directlycited by 0
  14. 2025A Unified Framework for Zero-Shot Reinforcement Learningcited by 0
  15. 2025Improved Offline Reinforcement Learning via Quantum Metric Encodingcited by 0
  16. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  17. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0
  18. 2026DSD: Learning Diverse and Reusable Motor Skills via Diffusion Skill Discoverycited by 0
  19. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Foundation Policies with Hilbert Representations
  2. 2023Zero-Shot Reinforcement Learning from Low Quality Data
  3. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning
  4. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
  5. 2025Consistent Zero-Shot Imitation with Contrastive Goal Inference
  6. 2026Towards Zero-Shot Task Transfer with Neurosymbolic World Models
  7. 2026From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
  8. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning