# EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views

arXiv: https://arxiv.org/abs/2405.13659 · HTML: https://mlsanity.com/paper/2405.13659/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 90 of its 115 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 30 of 90)

- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation](https://mlsanity.com/paper/2301.07525/) (2023)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)

## Built on by (most cited first, 26 of 26)

- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data](https://mlsanity.com/paper/2507.07095/) (2025)
- [GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding](https://mlsanity.com/paper/2411.19626/) (2024)
- [Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model](https://mlsanity.com/paper/2508.06206/) (2025)
- [PICO: Reconstructing 3D People In Contact with Objects](https://mlsanity.com/paper/2504.17695/) (2025)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [HERO: Human Reaction Generation from Videos](https://mlsanity.com/paper/2503.08270/) (2025)
- [SIGMAN:Scaling 3D Human Gaussian Generation with Millions of Assets](https://mlsanity.com/paper/2504.06982/) (2025)
- [Grounding 3D Scene Affordance From Egocentric Interactions](https://mlsanity.com/paper/2409.19650/) (2024)
- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)
- [TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactions](https://mlsanity.com/paper/2510.14874/) (2025)
- [Task-Aware 3D Affordance Segmentation via 2D Guidance and Geometric Refinement](https://mlsanity.com/paper/2511.11702/) (2025)
- [ECHO: Ego-Centric modeling of Human-Object interactions](https://mlsanity.com/paper/2508.21556/) (2025)
- [Gloria: Consistent Character Video Generation via Content Anchors](https://mlsanity.com/paper/2603.29931/) (2026)
- [Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge](https://mlsanity.com/paper/2510.08316/) (2025)
- [Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span](https://mlsanity.com/paper/2511.18470/) (2025)
- [Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation](https://mlsanity.com/paper/2606.30598/) (2026)
- [HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding](https://mlsanity.com/paper/2603.02329/) (2026)
- [EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning](https://mlsanity.com/paper/2603.09731/) (2026)
- [End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction](https://mlsanity.com/paper/2603.14435/) (2026)
- [LogiShot: Logically Coherent Cross-Shot Video Generation](https://mlsanity.com/paper/2608.08820/) (2026)
- [Leverage Task Context for Object Affordance Ranking](https://mlsanity.com/paper/2411.16082/) (2024)
- [EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation](https://mlsanity.com/paper/2604.01421/) (2026)
- [IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusion](https://mlsanity.com/paper/2606.28604/) (2026)
- [EgoHRV: Continuous Heart Rate Variability Estimation from Egocentric Systems for Autonomic Response and Skill Assessment](https://mlsanity.com/paper/2608.18711/) (2026)
- [G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding](https://mlsanity.com/paper/2608.20157/) (2026)

## Abstract (arXiv)

Understanding egocentric human-object interaction (HOI) is a fundamental aspect of human-centric perception, facilitating applications like AR/VR and embodied AI. For the egocentric HOI, in addition to perceiving semantics e.g., ''what'' interaction is occurring, capturing ''where'' the interaction specifically manifests in 3D space is also crucial, which links the perception and operation. Existing methods primarily leverage observations of HOI to capture interaction regions from an exocentric view. However, incomplete observations of interacting parties in the egocentric view introduce ambiguity between visual observations and interaction contents, impairing their efficacy. From the egocentric view, humans integrate the visual cortex, cerebellum, and brain to internalize their intentions and interaction concepts of objects, allowing for the pre-formulation of interactions and making behaviors even when interaction regions are out of sight. In light of this, we propose harmonizing the visual appearance, head motion, and 3D object to excavate the object interaction concept and subject intention, jointly inferring 3D human contact and object affordance from egocentric videos. To achieve this, we present EgoChoir, which links object structures with interaction contexts inherent in appearance and head motion to reveal object affordance, further utilizing it to model human contact. Additionally, a gradient modulation is employed to adopt appropriate clues for capturing interaction regions across various egocentric scenarios. Moreover, 3D contact and affordance are annotated for egocentric videos collected from Ego-Exo4D and GIMO to support the task. Extensive experiments on them demonstrate the effectiveness and superiority of EgoChoir.