mlsanityData to 7 Oct 2026

Compositional Foundation Models for Hierarchical Planning

arXiv 2309.08587 ↗First posted 15 Sep 2023cs.LG cs.AI cs.ROcited by 92

Not on the map. Why?

Cited by 21 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

27 of 39 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
15 more
  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  3. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  4. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  5. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  6. 2022Fast Sampling of Diffusion Models via Operator Learningcited by 173
  7. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  8. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  9. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  10. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  11. 2018Zero-Shot Visual Imitationcited by 114
  12. 2021Skill Induction and Planning with Latent Languagecited by 99
  13. 2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behaviorcited by 33
  14. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  15. 2023Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regressioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 37; 2025: 34; 2026: 19.

92 papers.

Sort:

By area:

  1. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  3. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  4. 2023Video Language Planningcited by 160
  5. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  6. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  7. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  8. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  9. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  10. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  11. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  12. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
38 more
  1. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  2. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  3. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  4. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  5. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  6. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  7. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  8. 2024Compositional Generative Modeling: A Single Model is Not All You Needcited by 41
  9. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  10. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  11. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  12. 2024JUICER: Data-Efficient Imitation Learning for Robotic Assemblycited by 36
  13. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  14. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  15. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  16. 2025GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulationcited by 26
  17. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 25
  18. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 25
  19. 2024VideoAgent: Self-Improving Video Generationcited by 24
  20. 2025A Survey of Interactive Generative Videocited by 23
  21. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  22. 2024Compositional Generative Inverse Designcited by 20
  23. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  24. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  25. 2024Generative Image as Action Modelscited by 18
  26. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  27. 2025RNE: plug-and-play diffusion inference-time control and energy-based trainingcited by 16
  28. 2026S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresightcited by 16
  29. 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
  30. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  31. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  32. 2025Mechanisms of Projective Composition of Diffusion Modelscited by 13
  33. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  34. 2025Solving New Tasks by Adapting Internet Video Knowledgecited by 13
  35. 2025Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modelingcited by 13
  36. 2024Generative Hierarchical Materials Searchcited by 12
  37. 2024Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Controlcited by 11
  38. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Planning with Latent World Models
  2. 2026Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation
  3. 2026H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
  4. 2026H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model
  5. 2026Spatially Grounded Long-Horizon Task Planning in the Wild
  6. 2025MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment
  7. 2026HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System
  8. 2026Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation