mlsanityData to 14 Oct 2026

Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration

arXiv 2410.18076 ↗First posted 23 Oct 2024cs.LG cs.AI stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Unsupervised pretraining has been transformative in many supervised domains. However, applying such ideas to reinforcement learning (RL) presents a unique challenge in that fine-tuning does not involve mimicking task-specific data, but rather exploring and locating the solution through iterative self-improvement. In this work, we study how unlabeled offline trajectory data can be leveraged to learn efficient exploration strategies. While prior data can be used to pretrain a set of low-level skills, or as additional off-policy data for online RL, it has been unclear how to combine these ideas effectively for online exploration. Our method SUPE (Skills from Unlabeled Prior data for Exploration) demonstrates that a careful combination of these ideas compounds their benefits. Our method first extracts low-level skills using a variational autoencoder (VAE), and then pseudo-labels unlabeled trajectories with optimistic rewards and high-level action labels, transforming prior data into high-level, task-relevant examples that encourage novelty-seeking behavior. Finally, SUPE uses these transformed examples as additional off-policy data for online RL to learn a high-level policy that composes pretrained low-level skills to explore efficiently. In our experiments, SUPE consistently outperforms prior strategies across a suite of 42 long-horizon, sparse-reward tasks. Code: https://github.com/rail-berkeley/supe.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 73 of its 93 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

73 of 93 references.

References · 73

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
38 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  3. 2016The Option-Critic Architecturecited by 448
  4. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  5. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  6. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  8. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  9. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  10. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  11. 2016Variational Intrinsic Controlcited by 267
  12. 2017Count-Based Exploration with Neural Density Modelscited by 257
  13. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  14. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  15. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  16. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  17. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  18. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  19. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  20. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  21. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  22. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  23. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  24. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  25. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  26. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  27. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  28. 2018Variational Option Discovery Algorithmscited by 122
  29. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  30. 2022Jump-Start Reinforcement Learningcited by 112
  31. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  32. 2018Neural probabilistic motor primitives for humanoid controlcited by 97
  33. 2024Foundation Policies with Hilbert Representationscited by 94
  34. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  35. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  36. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  37. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  38. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Action Chunkingcited by 107
  2. 2026Q-learning with Adjoint Matchingcited by 37
  3. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
  4. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  5. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 5
  6. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 5
  7. 2025Learning Skills from Action-Free Videoscited by 3
  8. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  9. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedbackcited by 0
  10. 2026SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Spacecited by 0
  11. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  12. 2026Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learningcited by 0
1 more
  1. 2026Learning Transferable Skills using Goal-Conditioned Bisimulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  2. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning
  3. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  4. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
  5. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories
  6. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  7. 2022The Challenges of Exploration for Offline Reinforcement Learning
  8. 2026Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals