mlsanityData to 13 Oct 2026

Universal Humanoid Motion Representations for Physics-Based Control

arXiv 2310.04582 ↗First posted 6 Oct 2023cs.CV cs.GR cs.ROcited by 111

Not on the map. Why?

Abstract (arXiv)

We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids and the inherent difficulties in reinforcement learning, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representation space. To achieve this, we first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RL, we show that our policies solve tasks using human-like behavior. We demonstrate the effectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers.

Key facts. Cited by 111 papers in the map (data through 13 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

39 of 55 references.

References · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2017Neural Discrete Representation Learningcited by 4,680
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,213
  5. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,195
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 890
  7. 2023MotionGPT: Human Motion as a Foreign Languagecited by 525
  8. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 498
  9. 2015Policy Distillationcited by 423
  10. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 399
  11. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  12. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
27 more
  1. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 321
  2. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 201
  3. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  4. 2021Character Controllers Using Motion VAEscited by 183
  5. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  6. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  7. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 138
  8. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 113
  9. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  10. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  11. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  12. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 98
  13. 2018Neural probabilistic motor primitives for humanoid controlcited by 96
  14. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 91
  15. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
  16. 2018Kickstarting Deep Reinforcement Learningcited by 77
  17. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  18. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
  19. 2020Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesiscited by 63
  20. 2021Estimating Egocentric 3D Human Pose in Global Spacecited by 56
  21. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 39
  22. 2021Discrete-Valued Neural Communicationcited by 39
  23. 2023Neural Categorical Priors for Physics-Based Character Controlcited by 37
  24. 2021Task-Generic Hierarchical Human Motion Prior using VAEscited by 33
  25. 2022Embodied Scene-aware Human Pose Estimationcited by 28
  26. 2022Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviorscited by 27
  27. 2023DiffMimic: Efficient Motion Mimicking with Differentiable Physicscited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 21; 2025: 48; 2026: 42.

111 papers.

Sort:

By area:

  1. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 169
  2. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 132
  3. 2024Expressive Whole-Body Control for Humanoid Robotscited by 118
  4. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 113
  5. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 106
  6. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 103
  7. 2025Flow Matching Policy Gradientscited by 90
  8. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 88
  9. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 82
  10. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 76
  11. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 68
  12. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 67
38 more
  1. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 58
  2. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 52
  3. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 48
  4. 2024Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Controlcited by 41
  5. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 40
  6. 2024MaskControl: Spatio-Temporal Control for Masked Motion Synthesiscited by 38
  7. 2024LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environmentcited by 33
  8. 2024PDP: Physics-Based Character Animation via Diffusion Policycited by 30
  9. 2024SMPLOlympics: Sports Environments for Physically Simulated Humanoidscited by 29
  10. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 27
  11. 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Controlcited by 27
  12. 2024PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarioscited by 24
  13. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 24
  14. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 22
  15. 2025Learning Getting-Up Policies for Real-World Humanoid Robotscited by 21
  16. 2025Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesiscited by 20
  17. 2025PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction Systemcited by 18
  18. 2024Real-Time Simulated Avatar from Head-Mounted Sensorscited by 17
  19. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  20. 2025DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human Referencescited by 14
  21. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusioncited by 14
  22. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  23. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  24. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 11
  25. 2025KungfuBot2: Learning Versatile Motion Skills for Humanoid Whole-Body Controlcited by 11
  26. 2025Text2Interact: High-Fidelity and Diverse Text-to-Two-Person Interaction Generationcited by 11
  27. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  28. 2025GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skillcited by 9
  29. 2025SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulationcited by 9
  30. 2025SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blendingcited by 9
  31. 2025ModSkill: Physical Character Skill Modularizationcited by 7
  32. 2025KINESIS: Motion Imitation for Human Musculoskeletal Locomotioncited by 7
  33. 2025Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputscited by 6
  34. 2025PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstructioncited by 6
  35. 2026ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generationcited by 6
  36. 2024A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motionscited by 5
  37. 2025Free-T2M: Robust Text-to-Motion Generation for Humanoid Robots via Frequency-Domaincited by 5
  38. 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learningcited by 5

50 most cited of 111 All 111 (JSON)