mlsanityData to 7 Oct 2026

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors

arXiv 2103.17265 ↗First posted 31 Mar 2021cs.CVcited by 137

Not on the map. Why?

Abstract (arXiv)

We introduce (HPS) Human POSEitioning System, a method to recover the full 3D pose of a human registered with a 3D scan of the surrounding environment using wearable sensors. Using IMUs attached at the body limbs and a head mounted camera looking outwards, HPS fuses camera based self-localization with IMU-based human body tracking. The former provides drift-free but noisy position and orientation estimates while the latter is accurate in the short-term but subject to drift over longer periods of time. We show that our optimization-based integration exploits the benefits of the two, resulting in pose accuracy free of drift. Furthermore, we integrate 3D scene constraints into our optimization, such as foot contact with the ground, resulting in physically plausible motion. HPS complements more common third-person-based 3D pose estimation methods. It allows capturing larger recording volumes and longer periods of motion, and could be used for VR/AR applications where humans interact with the scene without requiring direct line of sight with an external camera, or to train agents that navigate and interact with the environment based on first-person visual input, like real humans. With HPS, we recorded a dataset of humans interacting with large 3D scenes (300-1000 sq.m) consisting of 7 subjects and more than 3 hours of diverse motion. The dataset, code and video will be available on the project page: http://virtualhumans.mpi-inf.mpg.de/hps/ .

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 37 of its 80 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

37 of 80 references.

References · 37

  1. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  2. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
  5. 2017End-to-end Recovery of Human Shape and Posecited by 883
  6. 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 556
  7. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  8. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  9. 2019Multi-Garment Net: Learning to Dress 3D People from Imagescited by 244
  10. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  11. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  12. 2019Understanding the Limitations of CNN-based Absolute Camera Pose Regressioncited by 177
25 more
  1. 2020Visual Camera Re-Localization from RGB and RGB-D Images Using DSACcited by 176
  2. 2019Tex2Shape: Detailed Full Human Body Geometry From a Single Imagecited by 166
  3. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 151
  4. 2020TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Stylecited by 148
  5. 2016Image-based localization using LSTMs for structured feature correlationcited by 136
  6. 2020Combining Implicit Function Learning and Parametric Models for 3D Human Reconstructioncited by 133
  7. 2019Generating 3D People in Scenes without Peoplecited by 127
  8. 2016PlaNet - Photo Geolocation with Convolutional Neural Networkscited by 114
  9. 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 108
  10. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  11. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  12. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72
  13. 2019Expert Sample Consensus Applied to Camera Re-Localizationcited by 70
  14. 2018Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Cameracited by 67
  15. 2020Learning to Transfer Texture from Clothing Images to 3D Humanscited by 58
  16. 2020SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Cameracited by 49
  17. 2016Going Deeper into First-Person Activity Recognitioncited by 48
  18. 2019Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Videocited by 48
  19. 2016Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Videocited by 45
  20. 2018Binge Watching: Scaling Affordance Learning from Sitcomscited by 41
  21. 2019SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representationscited by 37
  22. 2018Real-Time RGB-D Camera Pose Estimation in Novel Scenes using a Relocalisation Cascadecited by 35
  23. 2020MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimationcited by 35
  24. 2018Leveraging Deep Visual Descriptors for Hierarchical Efficient Localizationcited by 29
  25. 2016Pose for Action - Action for Posecited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 19; 2023: 29; 2024: 40; 2025: 26; 2026: 16.

137 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  3. 2023Object Motion Guided Human Motion Synthesiscited by 206
  4. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  5. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  6. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
  7. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  8. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  9. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  10. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  11. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  12. 2023Controllable Human-Object Interaction Synthesiscited by 107
38 more
  1. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 105
  2. 2023Human Motion Generation: A Surveycited by 95
  3. 2023EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wildcited by 95
  4. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
  5. 2022Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fieldscited by 85
  6. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  7. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  8. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  9. 2022Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generationcited by 75
  10. 2022MeshLoc: Mesh-Based Visual Localizationcited by 62
  11. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  12. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  13. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  14. 2022FLAG: Flow-based 3D Avatar Generation from Sparse Observationscited by 54
  15. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  16. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  17. 2023EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensorscited by 48
  18. 2023SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environmentscited by 47
  19. 2022NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactionscited by 44
  20. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Modelscited by 44
  21. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  22. 2023Generating Continual Human Motion in Diverse 3D Scenescited by 39
  23. 2022HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDARcited by 38
  24. 2021Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactionscited by 37
  25. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  26. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 36
  27. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
  28. 2023Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Viewscited by 33
  29. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 33
  30. 2024LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environmentcited by 33
  31. 2023EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywherecited by 31
  32. 2022The One Where They Reconstructed 3D Humans and Environments in TV Showscited by 30
  33. 2022Scene-aware Egocentric 3D Human Pose Estimationcited by 30
  34. 2024Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Cameracited by 30
  35. 2023Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensorscited by 29
  36. 2024WHAC: World-grounded Humans and Camerascited by 29
  37. 2022Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motioncited by 27
  38. 2023CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactionscited by 27

50 most cited of 137 All 137 (JSON)