mlsanityData to 14 Oct 2026

QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars

arXiv 2209.09391 ↗First posted 20 Sep 2022cs.CV cs.GRcited by 77

Not on the map. Why?

Abstract (arXiv)

Real-time tracking of human body motion is crucial for interactive and immersive experiences in AR/VR. However, very limited sensor data about the body is available from standalone wearable devices such as HMDs (Head Mounted Devices) or AR glasses. In this work, we present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers, and simulates plausible and physically valid full body motions. Using high quality full body motion as dense supervision during training, a simple policy network can learn to output appropriate torques for the character to balance, walk, and jog, while closely following the input signals. Our results demonstrate surprisingly similar leg motions to ground truth without any observations of the lower body, even when the input is only the 6D transformations of the HMD. We also show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 21 of its 51 references that are in the map. First posted on arXiv 20 Sep 2022.

Builds on

21 of 51 references.

References · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  5. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 575
  6. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  7. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 324
  8. 2018Learning 3D Human Dynamics from Videocited by 316
  9. 2021Robust Motion In-betweeningcited by 229
  10. 2021Character Controllers Using Motion VAEscited by 184
  11. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
  12. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 163
9 more
  1. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 146
  2. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 110
  3. 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 109
  4. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 108
  5. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  6. 2019DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Comparecited by 100
  7. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  8. 2022Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generationcited by 76
  9. 2022FLAG: Flow-based 3D Avatar Generation from Sparse Observationscited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 21; 2024: 27; 2025: 15; 2026: 11.

77 papers.

Sort:

By area:

  1. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
  2. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
  3. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 130
  4. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 116
  5. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 116
  6. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 107
  7. 2023EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wildcited by 97
  8. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 94
  9. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 94
  10. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 89
  11. 2022GFPose: Learning 3D Human Pose Prior with Gradient Fieldscited by 57
  12. 2023EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensorscited by 48
38 more
  1. 2023BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesiscited by 45
  2. 2023Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modelingcited by 43
  3. 2022Learning to Imitate Object Interactions from Internet Videoscited by 34
  4. 2023EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywherecited by 34
  5. 2023DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusioncited by 31
  6. 2024Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Cameracited by 31
  7. 2023QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensorscited by 28
  8. 2024HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observationscited by 25
  9. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 23
  10. 2023SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Datacited by 21
  11. 2024Stratified Avatar Generation from Sparse Observationscited by 21
  12. 2023HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observationscited by 18
  13. 2023Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capturecited by 18
  14. 2024PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitationcited by 18
  15. 2024MultiPhys: Multi-Person Physics-aware 3D Motion Estimationcited by 18
  16. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  17. 2024Real-Time Simulated Avatar from Head-Mounted Sensorscited by 17
  18. 2024SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillationcited by 17
  19. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  20. 2024A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signalscited by 16
  21. 2024EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUscited by 13
  22. 2024SimpleEgo: Predicting Probabilistic Body Pose from Egocentric Camerascited by 12
  23. 2024DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackerscited by 11
  24. 2024EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modelingcited by 11
  25. 2023Physics-based Motion Retargeting from Sparse Inputscited by 10
  26. 2023HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using Inertial Sensingcited by 8
  27. 2024Diffusion-based Human Motion Style Transfer with Semantic Guidancecited by 7
  28. 2025ModSkill: Physical Character Skill Modularizationcited by 7
  29. 2025PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstructioncited by 7
  30. 2023MOVIN: Real-time Motion Capture using a Single LiDARcited by 6
  31. 2024Hand-Object Interaction Controller (HOIC): Deep Reinforcement Learning for Reconstructing Interactions with Physicscited by 6
  32. 2025EventEgo3D++: 3D Human Motion Capture from a Head-Mounted Event Cameracited by 6
  33. 2024A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motionscited by 5
  34. 2025FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Videocited by 5
  35. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 5
  36. 2023Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditionscited by 3
  37. 2023Privacy Preservation in Artificial Intelligence and Extended Reality (AI-XR) Metaverses: A Surveycited by 3
  38. 2024Self-Avatar Animation in Virtual Reality: Impact of Motion Signals Artifacts on the Full-Body Pose Reconstructioncited by 3

50 most cited of 77 All 77 (JSON)