mlsanityData to 14 Oct 2026

Choreographer: Learning and Adapting Skills in Imagination

arXiv 2211.13350 ↗First posted 23 Nov 2022cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Unsupervised skill learning aims to learn a rich repertoire of behaviors without external supervision, providing artificial agents with the ability to control and influence the environment. However, without appropriate knowledge and exploration, skills may provide control only over a restricted area of the environment, limiting their applicability. Furthermore, it is unclear how to leverage the learned skill behaviors for adapting to downstream tasks in a data-efficient manner. We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination. Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model. During adaptation, the agent uses a meta-controller to evaluate and adapt the learned skills efficiently by deploying them in parallel in imagination. Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy. The skills can be used to effectively adapt to downstream tasks, as we show in the URL benchmark, where we outperform previous approaches from both pixels and states inputs. The learned skills also explore the environment thoroughly, finding sparse rewards more frequently, as shown in goal-reaching tasks from the DMC Suite and Meta-World. Website and code: https://skillchoreographer.github.io/

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 57 of its 77 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

57 of 77 references.

References · 57

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  12. 2020Mastering Atari with Discrete World Modelscited by 1,007
38 more
  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2018DeepMind Control Suitecited by 956
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  4. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  9. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  10. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  11. 2019Learning Latent Plans from Playcited by 316
  12. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  13. 2016Variational Intrinsic Controlcited by 267
  14. 2019Self-Supervised Exploration via Disagreementcited by 265
  15. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  16. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  17. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  18. 2019Efficient Exploration via State Marginal Matchingcited by 188
  19. 2021Reinforcement Learning with Prototypical Representationscited by 179
  20. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  21. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  22. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  23. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  24. 2021Discovering and Achieving Goals via World Modelscited by 131
  25. 2018Variational Option Discovery Algorithmscited by 122
  26. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  27. 2021APS: Active Pretraining with Successor Featurescited by 112
  28. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 104
  29. 2022Deep Hierarchical Planning from Pixelscited by 101
  30. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  31. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
  32. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 76
  33. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
  34. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 62
  35. 2018Theory and Experiments on Vector Quantized Autoencoderscited by 55
  36. 2021Hierarchical Skills for Efficient Explorationcited by 38
  37. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  38. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 10; 2025: 10; 2026: 3.

31 papers.

Sort:

By area:

  1. 2024Behavior Generation with Latent Actionscited by 160
  2. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  3. 2026Hierarchical Planning with Latent World Modelscited by 43
  4. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 31
  5. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  6. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
  7. 2024Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Criticscited by 13
  8. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  9. 2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11
  10. 2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learningcited by 9
  11. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  12. 2023Learning to Discover Skills through Guidancecited by 8
19 more
  1. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 8
  2. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  3. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  4. 2024Dr. Strategy: Model-Based Generalist Agents with Strategic Dreamingcited by 6
  5. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  6. 2023Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreementcited by 4
  7. 2024Gradient-based inference of abstract task representations for generalization in neural networkscited by 4
  8. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learningcited by 3
  9. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  10. 2023Object-Centric Scene Representations using Active Inferencecited by 2
  11. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  12. 2025Information-Theoretic Policy Pre-Training with Empowermentcited by 2
  13. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  14. 2025Subgoal-Guided Policy Heuristic Search with Learned Subgoalscited by 1
  15. 2025Periodic Skill Discoverycited by 1
  16. 2026SUSD: Structured Unsupervised Skill Discovery through State Factorizationcited by 1
  17. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 1
  18. 2023Inferring Hierarchical Structure in Multi-Room Maze Environmentscited by 0
  19. 2025Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulation
  2. 2026Video2Skill: From Streaming Experience to Reusable Embodied Skills
  3. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptation
  4. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling
  5. 2026WISE: World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models
  6. 2026SkillWeaver: Agentic Exploration over Neural Interaction Skills for Scalable Robot Data Generation
  7. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play
  8. 2026UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy