mlsanityData to 14 Oct 2026

Variational Option Discovery Algorithms

arXiv 1807.10299 ↗First posted 26 Jul 2018cs.AIcited by 122

Not on the map. Why?

Cited by 21 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We explore methods for option discovery based on variational inference and make two algorithmic contributions. First: we highlight a tight connection between variational option discovery methods and variational autoencoders, and introduce Variational Autoencoding Learning of Options by Reinforcement (VALOR), a new method derived from the connection. In VALOR, the policy encodes contexts from a noise distribution into trajectories, and the decoder recovers the contexts from the complete trajectories. Second: we propose a curriculum learning approach where the number of contexts seen by the agent increases whenever the agent's performance is strong enough (as measured by the decoder) on the current set of contexts. We show that this simple trick stabilizes training for VALOR and prior variational option discovery methods, allowing a single agent to learn many more modes of behavior than it could with a fixed context distribution. Finally, we investigate other topics related to variational option discovery, including fundamental limitations of the general approach and the applicability of learned options to downstream tasks.

Key facts. Cited by 122 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 27; 2022: 26; 2023: 19; 2024: 16; 2025: 19; 2026: 13.

122 papers.

Sort:

By area:

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  3. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  4. 2021Discovering and Achieving Goals via World Modelscited by 131
  5. 2022Deep Hierarchical Planning from Pixelscited by 101
  6. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  7. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  8. 2021Reward is enough for convex MDPscited by 84
  9. 2023XSkill: Cross Embodiment Skill Discoverycited by 75
  10. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
  11. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 65
  12. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
38 more
  1. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  2. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  3. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  4. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  5. 2021Learning more skills through optimistic explorationcited by 38
  6. 2021Hierarchical Skills for Efficient Explorationcited by 38
  7. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
  8. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  9. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
  10. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
  11. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  12. 2021Wasserstein Unsupervised Reinforcement Learningcited by 22
  13. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  14. 2022Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motionscited by 21
  15. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
  16. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
  17. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
  18. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 20
  19. 2021Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reachingcited by 19
  20. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  21. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 17
  22. 2021Co-GAIL: Learning Diverse Strategies for Human-Robot Collaborationcited by 17
  23. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  24. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  25. 2021Discovering Diverse Athletic Jumping Strategiescited by 16
  26. 2021Learning Task Decomposition with Ordered Memory Policy Networkcited by 15
  27. 2021Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learningcited by 15
  28. 2021Constructing a Good Behavior Basis for Transfer using Generalized Policy Updatescited by 14
  29. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  30. 2021Adaptable Agent Populations via a Generative Model of Policiescited by 12
  31. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  32. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedbackcited by 12
  33. 2021Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximizationcited by 11
  34. 2022One After Another: Learning Incremental Skills for a Changing Worldcited by 10
  35. 2022MO2: Model-Based Offline Optionscited by 10
  36. 2022A Mixture of Surprises for Unsupervised Reinforcement Learningcited by 9
  37. 2023Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skillscited by 9
  38. 2021Average-Reward Learning and Planning with Optionscited by 8

50 most cited of 122 All 122 (JSON)