mlsanityData to 7 Oct 2026

Deep Hierarchical Planning from Pixels

arXiv 2206.04114 ↗First posted 8 Jun 2022cs.AI cs.LG cs.ROcited by 96

On the map: World models → Latent predictive world models

Abstract (arXiv)

Intelligent agents need to select long sequences of actions to solve complex tasks. While humans easily break down tasks into subgoals and reach them through millions of muscle commands, current artificial intelligence is limited to tasks with horizons of a few hundred decisions, despite large compute budgets. Research on hierarchical reinforcement learning aims to overcome this limitation but has proven to be challenging, current methods rely on manually specified goal spaces or subtasks, and no general solution exists. We introduce Director, a practical method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. The high-level policy maximizes task and exploration rewards by selecting latent goals and the low-level policy learns to achieve the goals. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director outperforms exploration methods on tasks with sparse rewards, including 3D maze traversal with a quadruped robot from an egocentric camera and proprioception, without access to the global position or top-down view that was used by prior work. Director also learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 42 of its 61 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

42 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Other references · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2020Mastering Atari with Discrete World Modelscited by 990
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2018DeepMind Control Suitecited by 935
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  11. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  12. 2020Planning to Explore via Self-Supervised World Modelscited by 365
29 more
  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  2. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  3. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  4. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  5. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  6. 2016Variational Intrinsic Controlcited by 262
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  8. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  9. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  10. 2021Discovering and Achieving Goals via World Modelscited by 131
  11. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  12. 2018Variational Option Discovery Algorithmscited by 121
  13. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  14. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  15. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  16. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  17. 2017Meta Learning Shared Hierarchiescited by 93
  18. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  19. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  20. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  21. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  22. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  23. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  24. 2021Learning more skills through optimistic explorationcited by 37
  25. 2021Hierarchical Skills for Efficient Explorationcited by 37
  26. 2020Action and Perception as Divergence Minimizationcited by 31
  27. 2018Hierarchical visuomotor control of humanoidscited by 29
  28. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  29. 2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 19; 2024: 35; 2025: 20; 2026: 18.

96 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  4. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  5. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  6. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  7. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 87
  8. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  9. 2024Simple Hierarchical Planning with Diffusioncited by 71
  10. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  11. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  12. 2026Hierarchical Planning with Latent World Modelscited by 34
38 more
  1. 2025Monte Carlo Tree Diffusion for System 2 Planningcited by 33
  2. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  3. 2023Planning Goals for Explorationcited by 28
  4. 2024Autoregressive Action Sequence Learning for Robotic Manipulationcited by 26
  5. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  6. 2023Gradient-based Planning with World Modelscited by 18
  7. 2024Privileged Sensing Scaffolds Reinforcement Learningcited by 18
  8. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  9. 2023Curious Replay for Model-based Adaptationcited by 16
  10. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  11. 2025GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulationcited by 15
  12. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
  13. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  14. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  15. 2025GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulationcited by 10
  16. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  17. 2023Combining Reconstruction and Contrastive Methods for Multimodal Representations in RLcited by 9
  18. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learningcited by 9
  19. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  20. 2023Hieros: Hierarchical Imagination on Structured State Space Sequence World Modelscited by 8
  21. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  22. 2025Flattening Hierarchies with Policy Bootstrappingcited by 8
  23. 2025Spatial-Temporal Aware Visuomotor Diffusion Policy Learningcited by 8
  24. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  25. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 8
  26. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
  27. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  28. 2025World Modeling with Probabilistic Structure Integrationcited by 7
  29. 2024Dr. Strategy: Model-Based Generalist Agents with Strategic Dreamingcited by 6
  30. 2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
  31. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  32. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  33. 2023Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learningcited by 4
  34. 2024Information-driven Affordance Discovery for Efficient Robotic Manipulationcited by 4
  35. 2024Learning Abstract World Model for Value-preserving Planning with Optionscited by 4
  36. 2024Boosting Soft Q-Learning by Boundingcited by 4
  37. 2024Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensorscited by 4
  38. 2024Identifying Selections for Unsupervised Subtask Discoverycited by 4

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Think before Go: Hierarchical Reasoning for Image-goal Navigation
  2. 2026H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
  3. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  4. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learning
  5. 2020Planning from Pixels using Inverse Dynamics Models
  6. 2020World Model as a Graph: Learning Latent Landmarks for Planning
  7. 2018Hierarchical Imitation and Reinforcement Learning
  8. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning