mlsanityData to 7 Oct 2026

Perpetual Humanoid Control for Real-time Simulated Avatars

arXiv 2305.06456 ↗First posted 10 May 2023cs.CV cs.GR cs.ROcited by 198

Not on the map. Why?

Abstract (arXiv)

We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e.g. pose estimates from video or generated from language) and unexpected falls. Our controller scales up to learning ten thousand motion clips without using any external stabilizing forces and learns to naturally recover from fail-state. Given reference motion, our controller can perpetually control simulated avatars without requiring resets. At its core, we propose the progressive multiplicative control policy (PMCP), which dynamically allocates new network capacity to learn harder and harder motion sequences. PMCP allows efficient scaling for learning from large-scale motion databases and adding new tasks, such as fail-state recovery, without catastrophic forgetting. We demonstrate the effectiveness of our controller by using it to imitate noisy poses from video-based pose estimators and language-based motion generators in a live and real-time multi-person avatar use case.

Key facts. Cited by 198 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

35 of 57 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  6. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  7. 2022Human Motion Diffusion Modelcited by 1,062
  8. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  12. 2022Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Trackingcited by 345
23 more
  1. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  2. 2018Progressive Feature Alignment for Unsupervised Domain Adaptationcited by 183
  3. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  4. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  5. 2019Cross-Domain Adaptation for Animal Pose Estimationcited by 108
  6. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  7. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  8. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 98
  9. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  10. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  11. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  12. 2020Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesiscited by 63
  13. 2022Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formatscited by 50
  14. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 39
  15. 2020MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimationcited by 35
  16. 2022Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capturecited by 31
  17. 2022Embodied Scene-aware Human Pose Estimationcited by 28
  18. 2022PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervisioncited by 25
  19. 2022MoCapAct: A Multi-Task Dataset for Simulated Humanoid Controlcited by 24
  20. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  21. 2018Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Controlcited by 12
  22. 2022Learning to Get Upcited by 11
  23. 2022From Universal Humanoid Control to Automatic Physically Valid Character Creationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 36; 2025: 79; 2026: 77.

198 papers.

Sort:

By area:

  1. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 167
  2. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 128
  3. 2024Expressive Whole-Body Control for Humanoid Robotscited by 117
  4. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 112
  5. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  6. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 104
  7. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  8. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  9. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  10. 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 86
  11. 2025Flow Matching Policy Gradientscited by 86
  12. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 83
38 more
  1. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  2. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  3. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  4. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  5. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
  6. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  7. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 57
  8. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  9. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  10. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 47
  11. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  12. 2025Humanoid-VLA: Towards Universal Humanoid Control with Visual Integrationcited by 43
  13. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  14. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 40
  15. 2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 39
  16. 2024MaskControl: Spatio-Temporal Control for Masked Motion Synthesiscited by 37
  17. 2024LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environmentcited by 33
  18. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  19. 2024PDP: Physics-Based Character Animation via Diffusion Policycited by 30
  20. 2024SMPLOlympics: Sports Environments for Physically Simulated Humanoidscited by 29
  21. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 27
  22. 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Controlcited by 27
  23. 2024SkillMimic: Learning Basketball Interaction Skills from Demonstrationscited by 25
  24. 2024PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarioscited by 24
  25. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  26. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 24
  27. 2025SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrationscited by 23
  28. 2023InterControl: Zero-shot Human Interaction Generation by Controlling Every Jointcited by 22
  29. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 22
  30. 2025Learning Getting-Up Policies for Real-World Humanoid Robotscited by 21
  31. 2024Scaling Large Motion Models with Million-Level Human Motionscited by 20
  32. 2025Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesiscited by 20
  33. 2024PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitationcited by 18
  34. 2023Generalized Animal Imitator: Agile Locomotion with Versatile Motion Priorcited by 17
  35. 2024Real-Time Simulated Avatar from Head-Mounted Sensorscited by 17
  36. 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidancecited by 17
  37. 2024Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robotscited by 16
  38. 2025From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robotscited by 16

50 most cited of 198 All 198 (JSON)