mlsanityData to 14 Oct 2026

EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling

arXiv 2412.10235 ↗First posted 13 Dec 2024cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Estimating full-body motion using the tracking signals of head and hands from VR devices holds great potential for various applications. However, the sparsity and unique distribution of observations present a significant challenge, resulting in an ill-posed problem with multiple feasible solutions (i.e., hypotheses). This amplifies uncertainty and ambiguity in full-body motion estimation, especially for the lower-body joints. Therefore, we propose a new method, EnvPoser, that employs a two-stage framework to perform full-body motion estimation using sparse tracking signals and pre-scanned environment from VR devices. EnvPoser models the multi-hypothesis nature of human motion through an uncertainty-aware estimation module in the first stage. In the second stage, we refine these multi-hypothesis estimates by integrating semantic and geometric environmental constraints, ensuring that the final motion estimation aligns realistically with both the environmental context and physical interactions. Qualitative and quantitative experiments on two public datasets demonstrate that our method achieves state-of-the-art performance, highlighting significant improvements in human motion estimation within motion-environment interaction scenarios.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 47 of its 56 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

47 of 56 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  5. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  6. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  7. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  8. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  9. 2021MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimationcited by 151
  10. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 146
  11. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 139
  12. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
35 more
  1. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 110
  2. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 107
  3. 2022DiffPose: Toward More Reliable 3D Pose Estimationcited by 100
  4. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  5. 2019Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Networkcited by 92
  6. 2021Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flowscited by 85
  7. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 82
  8. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  9. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 77
  10. 2022Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generationcited by 76
  11. 2022COAP: Compositional Articulated Occupancy of Peoplecited by 59
  12. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  13. 2022FLAG: Flow-based 3D Avatar Generation from Sparse Observationscited by 55
  14. 2023Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modelingcited by 43
  15. 2023EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywherecited by 34
  16. 2023Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Viewscited by 33
  17. 2023DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusioncited by 31
  18. 2024Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Cameracited by 31
  19. 2023Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensorscited by 30
  20. 2023QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensorscited by 28
  21. 2024HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observationscited by 25
  22. 2023SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Datacited by 21
  23. 2024Stratified Avatar Generation from Sparse Observationscited by 21
  24. 2023Pose-Oriented Transformer with Uncertainty-Guided Refinement for 2D-to-3D Human Pose Estimationcited by 19
  25. 2023HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observationscited by 18
  26. 2024A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signalscited by 16
  27. 2023Learning Human Mesh Recovery in 3D Scenescited by 13
  28. 2024EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUscited by 13
  29. 2023Narrator: Towards Natural Control of Human-Scene Interaction Generation via Relationship Reasoningcited by 10
  30. 2023Timestamp-supervised Wearable-based Activity Segmentation and Recognition with Contrastive Learning and Order-Preserving Optimal Transportcited by 10
  31. 2023Body Knowledge and Uncertainty Modeling for Monocular 3D Human Body Reconstructioncited by 8
  32. 2024Multimodal Sense-Informed Prediction of 3D Human Motionscited by 6
  33. 2024DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimizationcited by 3
  34. 2024SMGDiff: Soccer Motion Generation using diffusion probabilistic modelscited by 3
  35. 2023Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semanticscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisionscited by 9
  2. 2025RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequencecited by 7
  3. 2025mmDEAR: mmWave Point Cloud Density Enhancement for Accurate Human Body Reconstructioncited by 5
  4. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 5
  5. 2026Egocentric Visibility-Aware Human Pose Estimationcited by 1
  6. 2026A Two-Stage Motion-Aware Framework for mmWave-based Human Mesh Recoverycited by 1
  7. 2025Mem-MLP: Real-Time 3D Human Motion Generation from Sparse Inputscited by 0
  8. 2026GLU: Global-Local-Uncertainty Fusion for Scalable Spatiotemporal Reconstruction and Forecastingcited by 0
  9. 2026B'ezier Degradation Modeling for LiDAR-based Human Motion Capturecited by 0
  10. 2026Diverse Yet Consistent: Context-Guided Diffusion with Energy-Based Joint Refinement for Multi-Agent Motion Predictioncited by 0
  11. 2026IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusioncited by 0