mlsanityData to 7 Oct 2026

Social EgoMesh Estimation

arXiv 2411.04598 ↗First posted 7 Nov 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Accurately estimating the 3D pose of the camera wearer in egocentric video sequences is crucial to modeling human behavior in virtual and augmented reality applications. The task presents unique challenges due to the limited visibility of the user's body caused by the front-facing camera mounted on their head. Recent research has explored the utilization of the scene and ego-motion, but it has overlooked humans' interactive nature. We propose a novel framework for Social Egocentric Estimation of body MEshes (SEE-ME). Our approach is the first to estimate the wearer's mesh using only a latent probabilistic diffusion model, which we condition on the scene and, for the first time, on the social wearer-interactee interactions. Our in-depth study sheds light on when social interaction matters most for ego-mesh estimation; it quantifies the impact of interpersonal distance and gaze direction. Overall, SEE-ME surpasses the current best technique, reducing the pose estimation error (MPJPE) by 53%. The code is available at https://github.com/L-Scofano/SEEME.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 61 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

54 of 61 references.

References · 54

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2022Human Motion Diffusion Modelcited by 1,062
  6. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  7. 2017End-to-end Recovery of Human Shape and Posecited by 883
  8. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  9. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  10. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  11. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  12. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
38 more
  1. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  2. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  3. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  4. 2016Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Posecited by 220
  5. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  6. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 162
  7. 2021Probabilistic Modeling for Human Mesh Recoverycited by 152
  8. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
  9. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  10. 2017VNect: Real-time 3D Human Pose Estimation with a Single RGB Cameracited by 129
  11. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  12. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 107
  13. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  14. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  15. 2017Self-supervised Learning of Motion Capturecited by 97
  16. 2020Contact and Human Dynamics from Monocular Videocited by 94
  17. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
  18. 2019Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Networkcited by 92
  19. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  20. 2021Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flowscited by 84
  21. 2019You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactionscited by 82
  22. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  23. 2019xR-EgoPose: Egocentric 3D Human Pose from an HMD Cameracited by 78
  24. 2021Neural Monocular 3D Human Motion Capture with Physical Awarenesscited by 70
  25. 2021Physics-based Human Motion Estimation and Synthesis from Videoscited by 65
  26. 2019OpenVSLAM: A Versatile Visual SLAM Frameworkcited by 64
  27. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  28. 2022UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capturecited by 56
  29. 2015Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Videocited by 54
  30. 20203D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Datacited by 50
  31. 2020SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Cameracited by 49
  32. 2023EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensorscited by 48
  33. 2016Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Videocited by 45
  34. 2023Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modelingcited by 41
  35. 2022Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Videocited by 37
  36. 2023DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusioncited by 36
  37. 2023Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Viewscited by 33
  38. 2017Generating Multiple Diverse Hypotheses for Human 3D Pose Consistent with 2D Joint Detectionscited by 30

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World
  2. 2024Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data
  3. 2024Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Models
  4. 2024EgoNav: Egocentric Scene-aware Human Trajectory Prediction
  5. 2024Estimating Body and Hand Motion in an Ego-sensed World
  6. 2023Generative Proxemics: A Prior for 3D Social Interaction from Images
  7. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  8. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions