# AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidance

arXiv: https://arxiv.org/abs/2606.29209 · HTML: https://mlsanity.com/paper/2606.29209/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning](https://mlsanity.com/paper/2406.08858/) (2024)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots](https://mlsanity.com/paper/2410.21229/) (2024)
- [GMT: General Motion Tracking for Humanoid Whole-Body Control](https://mlsanity.com/paper/2506.14770/) (2025)
- [Spherical Latent Spaces for Stable Variational Autoencoders](https://mlsanity.com/paper/1808.10805/) (2018)
- [Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.07773/) (2024)
- [AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control](https://mlsanity.com/paper/2505.03738/) (2025)
- [FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control](https://mlsanity.com/paper/2505.22642/) (2025)
- [LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction](https://mlsanity.com/paper/2506.13751/) (2025)
- [CLONE: Closed-Loop Whole-Body Humanoid Teleoperation for Long-Horizon Tasks](https://mlsanity.com/paper/2506.08931/) (2025)
- [HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos](https://mlsanity.com/paper/2509.16757/) (2025)
- [BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2511.04131/) (2025)
- [Residual Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2106.08050/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning](https://mlsanity.com/paper/2504.21738/) (2025)
- [Deep Residual Reinforcement Learning](https://mlsanity.com/paper/1905.01072/) (2019)
- [TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer](https://mlsanity.com/paper/2311.03622/) (2023)
- [From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2506.12779/) (2025)
- [Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots](https://mlsanity.com/paper/2408.07295/) (2024)
- [Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition](https://mlsanity.com/paper/2502.18510/) (2025)
- [DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusion](https://mlsanity.com/paper/2509.14353/) (2025)

## Built on by (most cited first, 1 of 1)

- [Workhorse: Learning Robust Whole-Body Humanoid Loco-Manipulation from Human Data](https://mlsanity.com/paper/2610.09117/) (2026)

## Abstract (arXiv)

We present AnyBody, a unified whole-body humanoid controller driven by an arbitrary subset of body keypoints chosen at deploy time. Prior physics-based trackers either rely on expensive full-body motion capture and error-prone trajectory retargeting, which bottleneck scalable data collection and policy learning, or decompose upper- and lower-body control into separate hierarchical representations, sacrificing the coordinated whole-body motions that loco-manipulation requires. We close this gap by learning a single latent motion representation that any keypoint subset can address. To achieve this, we first train a privileged teacher tracker on a large unstructured motion corpus and distill it online into a deterministic encoder-decoder student whose latent space is a unit sphere. We then train a transformer keypoint encoder that admits any subset of body keypoints through masked self-attention, aligning it to the privileged latent. Additionally, we treat the frozen decoder as a motor prior and specialize downstream tasks with a lightweight residual corrector in the latent space. We demonstrate the effectiveness of AnyBody by tracking large-scale human motions from arbitrary keypoint subsets, free-form control, flexibly teleoperating, and learning downstream behaviors including locomotion, in-air writing, and obstacle-reach.