mlsanityData to 14 Oct 2026

Unsupervised Behavior Extraction via Random Intent Priors

arXiv 2310.18687 ↗First posted 28 Oct 2023cs.LG cs.AIcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reward-free data is abundant and contains rich prior knowledge of human behaviors, but it is not well exploited by offline reinforcement learning (RL) algorithms. In this paper, we propose UBER, an unsupervised approach to extract useful behaviors from offline reward-free datasets via diversified rewards. UBER assigns different pseudo-rewards sampled from a given prior distribution to different agents to extract a diverse set of behaviors, and reuse them as candidate policies to facilitate the learning of new tasks. Perhaps surprisingly, we show that rewards generated from random neural networks are sufficient to extract diverse and useful behaviors, some even close to expert ones. We provide both empirical and theoretical evidence to justify the use of random priors for the reward function. Experiments on multiple benchmarks showcase UBER's ability to learn effective and diverse behavior sets that enhance sample efficiency for online RL, outperforming existing baselines. By reducing reliance on human supervision, UBER broadens the applicability of RL to real-world scenarios with abundant reward-free data.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 63 of its 70 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

63 of 70 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023Survival Instinct in Offline Reinforcement Learning5 Jun
  2. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning27 Feb
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  4. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  5. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discovery2 Dec
  6. 2022The Role of Coverage in Online Reinforcement Learning9 Oct
  7. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  8. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  9. 2022On the Role of Discount Factor in Offline Reinforcement Learning7 Jun
  10. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  11. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  12. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
11 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning16 Sep
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  8. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  9. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
28 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  2. 2016Deep Exploration via Bootstrapped DQNcited by 527
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  4. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  5. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  6. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  7. 2020Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentationcited by 300
  8. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  9. 2019Self-Supervised Exploration via Disagreementcited by 265
  10. 2013Learning to Optimize Via Posterior Samplingcited by 256
  11. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  12. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
  13. 2021Reinforcement Learning with Prototypical Representationscited by 179
  14. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  15. 2016Generalization Properties of Learning with Random Featurescited by 164
  16. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  17. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  18. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  19. 2020Primal Wasserstein Imitation Learningcited by 121
  20. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  21. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  22. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  23. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
  24. 2019Wasserstein Adversarial Imitation Learningcited by 60
  25. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  26. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21
  27. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  28. 2022CUP: Critic-Guided Policy Reusecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 5; 2026: 1.

13 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Action Chunkingcited by 107
  2. 2024Foundation Policies with Hilbert Representationscited by 94
  3. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  4. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  5. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  6. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  7. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  8. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  9. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 5
  10. 2024Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learningcited by 3
  11. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 3
  12. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
1 more
  1. 2025Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Expressing Diverse Human Driving Behavior with Probabilistic Rewards and Online Inference
  2. 2020Semi-supervised reward learning for offline reinforcement learning
  3. 2019Learning Human Objectives by Evaluating Hypothetical Behavior
  4. 2023Accelerating Exploration with Unlabeled Prior Data
  5. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generation
  6. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  7. 2025Distributional Inverse Reinforcement Learning
  8. 2021Offline Preference-Based Apprenticeship Learning