mlsanityData to 7 Oct 2026

HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations

arXiv 2403.03561 ↗First posted 6 Mar 2024cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

It is especially challenging to achieve real-time human motion tracking on a standalone VR Head-Mounted Display (HMD) such as Meta Quest and PICO. In this paper, we propose HMD-Poser, the first unified approach to recover full-body motions using scalable sparse observations from HMD and body-worn IMUs. In particular, it can support a variety of input scenarios, such as HMD, HMD+2IMUs, HMD+3IMUs, etc. The scalability of inputs may accommodate users' choices for both high tracking accuracy and easy-to-wear. A lightweight temporal-spatial feature learning network is proposed in HMD-Poser to guarantee that the model runs in real-time on HMDs. Furthermore, HMD-Poser presents online body shape estimation to improve the position accuracy of body joints. Extensive experimental results on the challenging AMASS dataset show that HMD-Poser achieves new state-of-the-art results in both accuracy and real-time performance. We also build a new free-dancing motion dataset to evaluate HMD-Poser's on-device performance and investigate the performance gap between synthetic data and real-captured sensor data. Finally, we demonstrate our HMD-Poser with a real-time Avatar-driving application on a commercial HMD. Our code and free-dancing motion dataset are available https://pico-ai-team.github.io/hmd-poser

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  4. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  5. 2017End-to-end Recovery of Human Shape and Posecited by 883
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  7. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  8. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  9. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  10. 2018Learning 3D Human Dynamics from Videocited by 315
  11. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  12. 2022MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Videocited by 158
29 more
  1. 2021MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimationcited by 151
  2. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 144
  3. 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 108
  4. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 107
  5. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 104
  6. 20233D Human Pose Estimation via Intuitive Physicscited by 100
  7. 2018DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensorcited by 95
  8. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
  9. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  10. 2021Estimating Egocentric 3D Human Pose in Global Spacecited by 56
  11. 2022UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capturecited by 56
  12. 2022FLAG: Flow-based 3D Avatar Generation from Sparse Observationscited by 54
  13. 2021THUNDR: Transformer-based 3D HUmaN Reconstruction with Markerscited by 50
  14. 2023EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensorscited by 48
  15. 2023NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Human Pose and Shape Estimationcited by 45
  16. 2023Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modelingcited by 41
  17. 2021VoxelTrack: Multi-Person 3D Human Pose Estimation and Tracking in the Wildcited by 39
  18. 2022Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Videocited by 37
  19. 2019Multiview-Consistent Semi-Supervised Learning for 3D Human Pose Estimationcited by 31
  20. 2022Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capturecited by 31
  21. 2021SOMA: Solving Optical Marker-Based MoCap Automaticallycited by 30
  22. 2021RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Streamcited by 26
  23. 2023QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensorscited by 26
  24. 2022Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localizationcited by 25
  25. 2023SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Datacited by 20
  26. 2022Learning Variational Motion Prior for Video-based Motion Capturecited by 17
  27. 2023HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observationscited by 17
  28. 2023Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capturecited by 17
  29. 2021Real-time RGBD-based Extended Body Pose Estimationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  2. 2024EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUscited by 12
  3. 2024EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modelingcited by 11
  4. 2025From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Modelscited by 11
  5. 2024Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Datacited by 8
  6. 2025UMotion: Uncertainty-driven Human Motion Estimation from Inertial and Ultra-wideband Unitscited by 5
  7. 2025Human Motion Capture from Loose and Sparse Inertial Sensors with Garment-aware Diffusion Modelscited by 5
  8. 2025Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokenscited by 4
  9. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
  10. 2026Ground Reaction Inertial Poser: Physics-based Human Motion Capture from Sparse IMUs and Insole Pressure Sensorscited by 2
  11. 2025KineST: A Kinematics-guided Spatiotemporal State Space Model for Human Motion Tracking from Sparse Signalscited by 1
  12. 2026Egocentric Visibility-Aware Human Pose Estimationcited by 1
10 more
  1. 2026Towards Real-World Wearable Motion Reconstructioncited by 1
  2. 2025Mem-MLP: Real-Time 3D Human Motion Generation from Sparse Inputscited by 0
  3. 2025OmniEgoCap: Camera-Agnostic Sequence-Level Egocentric Motion Reconstructioncited by 0
  4. 2026EgoPoseVR: Spatiotemporal Multi-Modal Reasoning for Egocentric Full-Body Pose in Virtual Realitycited by 0
  5. 2026AtomicMotion: Learning Human Motion From Different Human Partscited by 0
  6. 2026MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observationscited by 0
  7. 2026IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusioncited by 0
  8. 2026EgoExoMoCap: Distributed Ego-Exo Human Motion Capturecited by 0
  9. 2026One Sensor, Whole Body - 3D Body Pose from a Single Consumer Earbud IMUcited by 0
  10. 2026Reliability-Gated Fusion of Consumer Head and Foot IMUs for Lower-Body 3D Posecited by 0