mlsanityData to 7 Oct 2026

Behavior Contrastive Learning for Unsupervised Skill Discovery

arXiv 2305.04477 ↗First posted 8 May 2023cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, unsupervised skill discovery aims to learn diverse skills without extrinsic rewards. Previous methods discover skills by maximizing the mutual information (MI) between states and skills. However, such an MI objective tends to learn simple and static skills and may hinder exploration. In this paper, we propose a novel unsupervised skill discovery method through contrastive learning among behaviors, which makes the agent produce similar behaviors for the same skill and diverse behaviors for different skills. Under mild assumptions, our objective maximizes the MI between different behaviors based on the same skill, which serves as an upper bound of the previous MI objective. Meanwhile, our method implicitly increases the state entropy to obtain better state coverage. We evaluate our method on challenging mazes and continuous control tasks. The results show that our method generates diverse and far-reaching skills, and also obtains competitive performance in downstream tasks compared to the state-of-the-art methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 49 of its 57 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

49 of 57 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2018DeepMind Control Suitecited by 935
37 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  3. 2018MINE: Mutual Information Neural Estimationcited by 680
  4. 2019On Variational Bounds of Mutual Informationcited by 550
  5. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  7. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  8. 2020Understanding the Behaviour of Contrastive Losscited by 397
  9. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  10. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  11. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 299
  12. 2019Self-Supervised Exploration via Disagreementcited by 263
  13. 2016Variational Intrinsic Controlcited by 262
  14. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  15. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  16. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  17. 2019Efficient Exploration via State Marginal Matchingcited by 186
  18. 2021Reinforcement Learning with Prototypical Representationscited by 179
  19. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  20. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  21. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  22. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  23. 2021APS: Active Pretraining with Successor Featurescited by 109
  24. 2020Deep Reinforcement and InfoMax Learningcited by 79
  25. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
  26. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  27. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  28. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  29. 2021Learning more skills through optimistic explorationcited by 37
  30. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  31. 2020Relative Variational Intrinsic Controlcited by 34
  32. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  33. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  34. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  35. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
  36. 2021Mutual Information State Intrinsic Controlcited by 18
  37. 2020Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 10; 2026: 6.

26 papers.

Sort:

By area:

  1. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  3. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  4. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  5. 2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learningcited by 9
  6. 2024Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learningcited by 9
  7. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  8. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  9. 2024Constrained Intrinsic Motivation for Reinforcement Learningcited by 6
  10. 2025Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustmentcited by 3
  11. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  12. 2025VendiRL: A Framework for Self-Supervised Reinforcement Learning of Diversely Diverse Skillscited by 2
14 more
  1. 2023On the Value of Myopic Behavior in Policy Reusecited by 1
  2. 2025RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teamingcited by 1
  3. 2025Learning Task-Agnostic Motifs to Capture the Continuous Nature of Animal Behaviorcited by 1
  4. 2025Periodic Skill Discoverycited by 1
  5. 2026Task-Aware Exploration via a Predictive Bisimulation Metriccited by 1
  6. 2026Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewardscited by 1
  7. 2023ComSD: Balancing Behavioral Quality and Diversity in Unsupervised Skill Discoverycited by 0
  8. 2024CURLing the Dream: Contrastive Representations for World Modeling in Reinforcement Learningcited by 0
  9. 2025AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversificationcited by 0
  10. 2025Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learningcited by 0
  11. 2026Group-Invariant Unsupervised Skill Discovery: Symmetry-aware Skill Representations for Generalizable Behaviorcited by 0
  12. 2026Learning Generalizable Skill Policy with Data-Efficient Unsupervised RLcited by 0
  13. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  14. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0