mlsanityData to 7 Oct 2026

Estimating Body and Hand Motion in an Ego-sensed World

arXiv 2410.03665 ↗First posted 4 Oct 2024cs.CV cs.AIcited by 56

Not on the map. Why?

Abstract (arXiv)

We present EgoAllo, a system for human motion estimation from a head-mounted device. Using only egocentric SLAM poses and images, EgoAllo guides sampling from a conditional diffusion model to estimate 3D body pose, height, and hand parameters that capture a device wearer's actions in the allocentric coordinate frame of the scene. To achieve this, our key insight is in representation: we propose spatial and temporal invariance criteria for improving model performance, from which we derive a head motion conditioning parameterization that improves estimation by up to 18%. We also show how the bodies estimated by our system can improve hand estimation: the resulting kinematic and temporal constraints can reduce world-frame errors in single-frame estimates by 40%. Project page: https://egoallo.github.io/

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 93 of its 122 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

93 of 122 references.

References · 93

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
38 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2017Deep Setscited by 1,394
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2016Group Equivariant Convolutional Networkscited by 1,154
  6. 2017End-to-end Recovery of Human Shape and Posecited by 883
  7. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  8. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  9. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  10. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  11. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  12. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  13. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  14. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  15. 2023Human Motion Diffusion as a Generative Priorcited by 354
  16. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  17. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  18. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  19. 2018Learning 3D Human Dynamics from Videocited by 315
  20. 2023Reconstructing Hands in 3D with Transformerscited by 298
  21. 2022EDGE: Editable Dance Generation From Musiccited by 284
  22. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  23. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  24. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  25. 2023Object Motion Guided Human Motion Synthesiscited by 206
  26. 2017Unite the People: Closing the Loop Between 3D and 2D Human Representationscited by 205
  27. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 201
  28. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  29. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  30. 2021Character Controllers Using Motion VAEscited by 182
  31. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  32. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  33. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  34. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  35. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
  36. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  37. 2020Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimationcited by 113
  38. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 25; 2026: 31.

56 papers.

Sort:

By area:

  1. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  2. 2025Cameras as Relative Positional Encodingcited by 94
  3. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 57
  4. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  5. 2025Predicting 4D Hand Trajectory from Monocular Videoscited by 19
  6. 2025Viser: Imperative, Web-based 3D Visualization in Pythoncited by 19
  7. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generationcited by 17
  8. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  9. 2025REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioningcited by 14
  10. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  11. 2025From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Modelscited by 11
  12. 2025LookOut: Real-World Humanoid Egocentric Navigationcited by 9
38 more
  1. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
  2. 2025EgoReAct: Egocentric Video-Driven 3D Human Reaction Generationcited by 7
  3. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  4. 2026WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videoscited by 7
  5. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
  6. 2025Seeing without Pixels: Perception from Camera Trajectoriescited by 4
  7. 2025Diffusion Forcing for Multi-Agent Interaction Sequence Modelingcited by 4
  8. 2025EgoCogNav: Cognition-aware Human Egocentric Navigationcited by 3
  9. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  10. 2025Photoreal Scene Reconstruction from an Egocentric Devicecited by 2
  11. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 2
  12. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  13. 2026EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VRcited by 2
  14. 2026Ground Reaction Inertial Poser: Physics-based Human Motion Capture from Sparse IMUs and Insole Pressure Sensorscited by 2
  15. 2026VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenescited by 2
  16. 2026MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervisioncited by 2
  17. 2025Event-based Egocentric Human Pose Estimation in Dynamic Environmentcited by 1
  18. 2025Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Imagescited by 1
  19. 2025Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reachcited by 1
  20. 2026Egocentric Visibility-Aware Human Pose Estimationcited by 1
  21. 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 1
  22. 2026Efficient Feature-Free Initialization for Monocular Visual-Inertial Systems Using a Feed-Forward 3D Modelcited by 1
  23. 2026Towards Real-World Wearable Motion Reconstructioncited by 1
  24. 2026ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Videocited by 1
  25. 2025HAGI++: Head-Assisted Gaze Imputation and Generationcited by 0
  26. 2025OmniEgoCap: Camera-Agnostic Sequence-Level Egocentric Motion Reconstructioncited by 0
  27. 2026AG-EgoPose: Spatially Anchored Residual Correction with Action Context for Monocular Egocentric 3D Pose Estimationcited by 0
  28. 2026RoSHI: A Versatile Robot-oriented Suit for Human Data In-the-Wildcited by 0
  29. 2026EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generationcited by 0
  30. 2026MotionGRPO: Overcoming Low Intra-Group Diversity in GRPO-Based Egocentric Motion Recoverycited by 0
  31. 2026EgoForce: Robust Online Egocentric Motion Reconstruction via Diffusion Forcingcited by 0
  32. 2026Map-Mono-Ego: Map-Grounded Global Human Pose Estimation from Monocular Egocentric Videocited by 0
  33. 2026EgoRelight: Egocentric Human Capture and Illumination Recovery for Relightable and Photoreal Avatar Renderingcited by 0
  34. 2026Ultra Diffusion Poser: Diffusion-Based Human Motion Tracking From Sparse Inertial Sensors and Ranging-Based Between-Sensor Distancescited by 0
  35. 2026ARAPDiffusion: ARAP Regularization for Diffusion-Based Deformable Shape Space Learningcited by 0
  36. 2026EgoPriMo: Egocentric Motion Generation for Interactive Humanoid Controlcited by 0
  37. 2026EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversalcited by 0
  38. 2026EgoExoMoCap: Distributed Ego-Exo Human Motion Capturecited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data
  2. 2024EgoNav: Egocentric Scene-aware Human Trajectory Prediction
  3. 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis
  4. 2026ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild
  5. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmark
  6. 2025EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations
  7. 2026FUSION: Full-Body Unified Motion Prior for Body and Hands via Diffusion
  8. 2026StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video