# Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs

arXiv: https://arxiv.org/abs/2502.05641 · HTML: https://mlsanity.com/paper/2502.05641/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes](https://mlsanity.com/paper/2012.05522/) (2020)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model](https://mlsanity.com/paper/2304.08577/) (2023)
- [Neural probabilistic motor primitives for humanoid control](https://mlsanity.com/paper/1811.11711/) (2018)
- [CALM: Conditional Adversarial Latent Models for Directable Virtual Characters](https://mlsanity.com/paper/2305.02195/) (2023)
- [Learning human behaviors from motion capture by adversarial imitation](https://mlsanity.com/paper/1707.02201/) (2017)
- [MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations](https://mlsanity.com/paper/2310.10198/) (2023)
- [C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters](https://mlsanity.com/paper/2309.11351/) (2023)
- [PADL: Language-Directed Physics-Based Character Control](https://mlsanity.com/paper/2301.13868/) (2023)
- [PMP: Learning to Physically Interact with Environments using Part-wise Motion Priors](https://mlsanity.com/paper/2305.03249/) (2023)
- [MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation](https://mlsanity.com/paper/2007.07227/) (2020)
- [Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning](https://mlsanity.com/paper/2309.11359/) (2023)
- [Composite Motion Learning with Task Control](https://mlsanity.com/paper/2305.03286/) (2023)
- [MoCapAct: A Multi-Task Dataset for Simulated Humanoid Control](https://mlsanity.com/paper/2208.07363/) (2022)
- [DiffMimic: Efficient Motion Mimicking with Differentiable Physics](https://mlsanity.com/paper/2304.03274/) (2023)
- [DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs](https://mlsanity.com/paper/2010.08891/) (2020)
- [A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control](https://mlsanity.com/paper/2105.10066/) (2021)
- [From Universal Humanoid Control to Automatic Physically Valid Character Creation](https://mlsanity.com/paper/2206.09286/) (2022)
- [H-GAP: Humanoid Control with a Generalist Planner](https://mlsanity.com/paper/2312.02682/) (2023)
- [Physically Plausible Animation of Human Upper Body from a Single Image](https://mlsanity.com/paper/2212.04741/) (2022)

## Built on by (most cited first, 6 of 6)

- [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://mlsanity.com/paper/2507.05419/) (2025)
- [Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward](https://mlsanity.com/paper/2506.03191/) (2025)
- [Self-Intersection-Aware 3D Human Motion Generation Using an Efficient Human Sphere Proxy](https://mlsanity.com/paper/2605.26744/) (2026)
- [No More Marching: Learning Humanoid Locomotion for Short-Range SE(2) Targets](https://mlsanity.com/paper/2508.14098/) (2025)
- [WATCH: World-aware Allied Trajectory and pose reConstruction for Camera and Human](https://mlsanity.com/paper/2509.04600/) (2025)
- [Switch-JustDance: Benchmarking Whole Body Motion Tracking Controllers Using a Commercial Console Game](https://mlsanity.com/paper/2511.17925/) (2025)

## Abstract (arXiv)

This work focuses on generating realistic, physically-based human behaviors from multi-modal inputs, which may only partially specify the desired motion. For example, the input may come from a VR controller providing arm motion and body velocity, partial key-point animation, computer vision applied to videos, or even higher-level motion goals. This requires a versatile low-level humanoid controller that can handle such sparse, under-specified guidance, seamlessly switch between skills, and recover from failures. Current approaches for learning humanoid controllers from demonstration data capture some of these characteristics, but none achieve them all. To this end, we introduce the Masked Humanoid Controller (MHC), a novel approach that applies multi-objective imitation learning on augmented and selectively masked motion demonstrations. The training methodology results in an MHC that exhibits the key capabilities of catch-up to out-of-sync input commands, combining elements from multiple motion sequences, and completing unspecified parts of motions from sparse multimodal input. We demonstrate these key capabilities for an MHC learned over a dataset of 87 diverse skills and showcase different multi-modal use cases, including integration with planning frameworks to highlight MHC's ability to solve new user-defined tasks without any finetuning.