# H-GAP: Humanoid Control with a Generalist Planner

arXiv: https://arxiv.org/abs/2312.02682 · HTML: https://mlsanity.com/paper/2312.02682/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)

## Built on by (most cited first, 11 of 11)

- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning](https://mlsanity.com/paper/2502.03275/) (2025)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [Hierarchical World Models as Visual Whole-Body Humanoid Controllers](https://mlsanity.com/paper/2405.18418/) (2024)
- [STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization](https://mlsanity.com/paper/2506.03863/) (2025)
- [Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs](https://mlsanity.com/paper/2502.05641/) (2025)
- [Return-Aligned Decision Transformer](https://mlsanity.com/paper/2402.03923/) (2024)
- [Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning](https://mlsanity.com/paper/2411.07760/) (2024)
- [Improving Zero-Shot Offline RL via Behavioral Task Sampling](https://mlsanity.com/paper/2604.25496/) (2026)
- [In-Context Planning with Latent Temporal Abstractions](https://mlsanity.com/paper/2602.18694/) (2026)
- [PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control](https://mlsanity.com/paper/2609.24840/) (2026)

## Abstract (arXiv)

Humanoid control is an important research challenge offering avenues for integration into human-centric infrastructures and enabling physics-driven humanoid animations. The daunting challenges in this field stem from the difficulty of optimizing in high-dimensional action spaces and the instability introduced by the bipedal morphology of humanoids. However, the extensive collection of human motion-captured data and the derived datasets of humanoid trajectories, such as MoCapAct, paves the way to tackle these challenges. In this context, we present Humanoid Generalist Autoencoding Planner (H-GAP), a state-action trajectory generative model trained on humanoid trajectories derived from human motion-captured data, capable of adeptly handling downstream control tasks with Model Predictive Control (MPC). For 56 degrees of freedom humanoid, we empirically demonstrate that H-GAP learns to represent and generate a wide range of motor behaviours. Further, without any learning from online interactions, it can also flexibly transfer these behaviors to solve novel downstream control tasks via planning. Notably, H-GAP excels established MPC baselines that have access to the ground truth dynamics model, and is superior or comparable to offline RL methods trained for individual tasks. Finally, we do a series of empirical studies on the scaling properties of H-GAP, showing the potential for performance gains via additional data but not computing. Code and videos are available at https://ycxuyingchen.github.io/hgap/.