mlsanityData to 7 Oct 2026

URLB: Unsupervised Reinforcement Learning Benchmark

arXiv 2110.15191 ↗First posted 28 Oct 2021cs.LG cs.AI cs.ROcited by 138

Not on the map. Why?

Abstract (arXiv)

Deep Reinforcement Learning (RL) has emerged as a powerful paradigm to solve a range of complex yet specific control tasks. Yet training generalist agents that can quickly adapt to new tasks remains an outstanding challenge. Recent advances in unsupervised RL have shown that pre-training RL agents with self-supervised intrinsic rewards can result in efficient adaptation. However, these algorithms have been hard to compare and develop due to the lack of a unified benchmark. To this end, we introduce the Unsupervised Reinforcement Learning Benchmark (URLB). URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. Building on the DeepMind Control Suite, we provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods. We find that the implemented baselines make progress but are not able to solve URLB and propose directions for future research.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 54 of its 73 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

54 of 73 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2013Auto-Encoding Variational Bayescited by 3,980
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
38 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2018DeepMind Control Suitecited by 935
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  13. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  14. 2019Solving Rubik's Cube with a Robot Handcited by 699
  15. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  16. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  17. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  18. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  19. 2020Reinforcement Learning with Augmented Datacited by 452
  20. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  21. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  22. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  23. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  24. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  25. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  26. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  27. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  28. 2019Self-Supervised Exploration via Disagreementcited by 263
  29. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  30. 2016Variational Intrinsic Controlcited by 262
  31. 2020First return, then explorecited by 256
  32. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
  33. 2019Efficient Exploration via State Marginal Matchingcited by 186
  34. 2021Reinforcement Learning with Prototypical Representationscited by 179
  35. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  36. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  37. 2018Variational Option Discovery Algorithmscited by 121
  38. 2021APS: Active Pretraining with Successor Featurescited by 109

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 32; 2023: 33; 2024: 28; 2025: 26; 2026: 17.

138 papers.

Sort:

By area:

  1. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  3. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  4. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  5. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  6. 2024Foundation Policies with Hilbert Representationscited by 89
  7. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  8. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  9. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
  10. 2024Simple Hierarchical Planning with Diffusioncited by 71
  11. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  12. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
38 more
  1. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  2. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  3. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  4. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  5. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  6. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  7. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  8. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  9. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 29
  10. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  11. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  12. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  13. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  14. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  15. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  16. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  17. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  18. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  19. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 20
  20. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  21. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  22. 2023FOCUS: Object-Centric World Models for Robotics Manipulationcited by 17
  23. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  24. 2023Curious Replay for Model-based Adaptationcited by 16
  25. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  26. 2024IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoningcited by 16
  27. 2022Light-weight probing of unsupervised representations for Reinforcement Learningcited by 15
  28. 2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
  29. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  30. 2023Reward-Free Curricula for Training Robust World Modelscited by 13
  31. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  32. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  33. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12
  34. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  35. 2022k-Means Maximum Entropy Explorationcited by 11
  36. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  37. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  38. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11

50 most cited of 138 All 138 (JSON)