mlsanityData to 14 Oct 2026

Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skills

arXiv 2310.19424 ↗First posted 30 Oct 2023cs.LG cs.AI cs.ROcited by 9

Not on the map. Why?

Abstract (arXiv)

Mutual information-based reinforcement learning (RL) has been proposed as a promising framework for retrieving complex skills autonomously without a task-oriented reward function through mutual information (MI) maximization or variational empowerment. However, learning complex skills is still challenging, due to the fact that the order of training skills can largely affect sample efficiency. Inspired by this, we recast variational empowerment as curriculum learning in goal-conditioned RL with an intrinsic reward function, which we name Variational Curriculum RL (VCRL). From this perspective, we propose a novel approach to unsupervised skill discovery based on information theory, called Value Uncertainty Variational Curriculum (VUVC). We prove that, under regularity conditions, VUVC accelerates the increase of entropy in the visited states compared to the uniform curriculum. We validate the effectiveness of our approach on complex navigation and robotic manipulation tasks in terms of sample efficiency and state coverage speed. We also demonstrate that the skills discovered by our method successfully complete a real-world robot navigation task in a zero-shot setup and that incorporating these skills with a global planner further increases the performance.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

48 of 63 references.

References · 48

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2016Least Squares Generative Adversarial Networkscited by 1,001
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  11. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  12. 2016Variational Intrinsic Controlcited by 267
36 more
  1. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  2. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  3. 2017Count-Based Exploration with Neural Density Modelscited by 257
  4. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  5. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  6. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  7. 2019Efficient Exploration via State Marginal Matchingcited by 188
  8. 2021Reinforcement Learning with Prototypical Representationscited by 179
  9. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  10. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  11. 2019Goal-conditioned Imitation Learningcited by 162
  12. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  13. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  14. 2021Discovering and Achieving Goals via World Modelscited by 131
  15. 2018Variational Option Discovery Algorithmscited by 122
  16. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  17. 2018Count-Based Exploration with the Successor Representationcited by 112
  18. 2021APS: Active Pretraining with Successor Featurescited by 112
  19. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  20. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  21. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  22. 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 88
  23. 2020Automatic Curriculum Learning through Value Disagreementcited by 87
  24. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 78
  25. 2013Empowerment -- an Introductioncited by 68
  26. 2019Exploration via Hindsight Goal Generationcited by 54
  27. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  28. 2020Self-Paced Deep Reinforcement Learningcited by 50
  29. 2018Learning Curriculum Policies for Reinforcement Learningcited by 35
  30. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 34
  31. 2019Competitive Experience Replaycited by 23
  32. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 17
  33. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  34. 2018Accuracy-based Curriculum Learning in Deep Reinforcement Learningcited by 15
  35. 2021All-in-One: A DRL-based Control Switch Combining State-of-the-art Navigation Plannerscited by 8
  36. 2019Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2025Maximizing Confidence Alone Improves Reasoningcited by 100
  2. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  3. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 13
  4. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  5. 2025Consistent Zero-Shot Imitation with Contrastive Goal Inferencecited by 1
  6. 2025Imagine Beyond! Distributionally Robust Auto-Encoding for State Space Coverage in Online Reinforcement Learningcited by 0
  7. 2025Categorical Policies: Multimodal Policy Learning and Exploration in Continuous Controlcited by 0
  8. 2026Self-Curriculum Model-based Reinforcement Learning for Shape Control of Deformable Linear Objectscited by 0
  9. 2026Curriculum reinforcement learning with measurable task representation learningcited by 0