# Hierarchical World Models as Visual Whole-Body Humanoid Controllers

arXiv: https://arxiv.org/abs/2405.18418 · HTML: https://mlsanity.com/paper/2405.18418/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Humanoid and legged locomotion world models, one of 11 directions in World models.
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 28 May 2024.

## Placed in

- World models / [Humanoid and legged locomotion world models](https://mlsanity.com/wm/humanoid-and-legged-locomotion-world-models/)

## Builds on (most cited first, 30 of 39)

- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation](https://mlsanity.com/paper/2403.04436/) (2024)
- [Model Predictive Path Integral Control using Covariance Variable Importance Sampling](https://mlsanity.com/paper/1509.01149/) (2015)
- [HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation](https://mlsanity.com/paper/2403.10506/) (2024)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [Neural probabilistic motor primitives for humanoid control](https://mlsanity.com/paper/1811.11711/) (2018)
- [MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control](https://mlsanity.com/paper/2205.13600/) (2022)
- [Learning human behaviors from motion capture by adversarial imitation](https://mlsanity.com/paper/1707.02201/) (2017)
- [Simple Hierarchical Planning with Diffusion](https://mlsanity.com/paper/2401.02644/) (2024)
- [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://mlsanity.com/paper/1909.10618/) (2019)
- [MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/2212.05698/) (2022)
- [On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline](https://mlsanity.com/paper/2212.05749/) (2022)
- [Skill-based Model-based Reinforcement Learning](https://mlsanity.com/paper/2207.07560/) (2022)
- [Finetuning Offline World Models in the Real World](https://mlsanity.com/paper/2310.16029/) (2023)

## Built on by (most cited first, 27 of 27)

- [GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control](https://mlsanity.com/paper/2412.11198/) (2024)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [A Survey: Learning Embodied Intelligence from Physical Simulators and World Models](https://mlsanity.com/paper/2507.00917/) (2025)
- [Hierarchical Planning with Latent World Models](https://mlsanity.com/paper/2604.03208/) (2026)
- [RISE: Self-Improving Robot Policy with Compositional World Model](https://mlsanity.com/paper/2602.11075/) (2026)
- [Whole-Body Conditioned Egocentric Video Prediction](https://mlsanity.com/paper/2506.21552/) (2025)
- [Learning Getting-Up Policies for Real-World Humanoid Robots](https://mlsanity.com/paper/2502.12152/) (2025)
- [UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Control](https://mlsanity.com/paper/2504.12540/) (2025)
- [Embodied AI: From LLMs to World Models](https://mlsanity.com/paper/2509.20021/) (2025)
- [Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking](https://mlsanity.com/paper/2412.17730/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models](https://mlsanity.com/paper/2602.18739/) (2026)
- [A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges](https://mlsanity.com/paper/2409.07569/) (2024)
- [SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blending](https://mlsanity.com/paper/2506.09366/) (2025)
- [PhyWorld: Physics-Faithful World Model for Video Generation](https://mlsanity.com/paper/2605.19242/) (2026)
- [HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Model](https://mlsanity.com/paper/2602.11758/) (2026)
- [Gazing at Rewards: Eye Movements as a Lens into Human and AI Decision-Making in Hybrid Visual Foraging](https://mlsanity.com/paper/2411.09176/) (2024)
- [DyMoDreamer: World Modeling with Dynamic Modulation](https://mlsanity.com/paper/2509.24804/) (2025)
- [Lifting Embodied World Models for Planning and Control](https://mlsanity.com/paper/2604.26182/) (2026)
- [Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control](https://mlsanity.com/paper/2601.21363/) (2026)
- [ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking](https://mlsanity.com/paper/2602.06445/) (2026)
- [Emotion-Conditioned Short-Horizon Human Pose Forecasting with a Lightweight Predictive World Model](https://mlsanity.com/paper/2604.23532/) (2026)
- [MuJoCo MPC for Humanoid Control: Evaluation on HumanoidBench](https://mlsanity.com/paper/2408.00342/) (2024)
- [Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning](https://mlsanity.com/paper/2605.13207/) (2026)
- [Neuro-Inspired Inverse Learning for Planning and Control](https://mlsanity.com/paper/2605.24152/) (2026)
- [World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories](https://mlsanity.com/paper/2610.01742/) (2026)
- [H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning](https://mlsanity.com/paper/2610.06805/) (2026)

## Abstract (arXiv)

Whole-body control for humanoids is challenging due to the high-dimensional nature of the problem, coupled with the inherent instability of a bipedal morphology. Learning from visual observations further exacerbates this difficulty. In this work, we explore highly data-driven approaches to visual whole-body humanoid control based on reinforcement learning, without any simplifying assumptions, reward design, or skill primitives. Specifically, we propose a hierarchical world model in which a high-level agent generates commands based on visual observations for a low-level agent to execute, both of which are trained with rewards. Our approach produces highly performant control policies in 8 tasks with a simulated 56-DoF humanoid, while synthesizing motions that are broadly preferred by humans.