# EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates

arXiv: https://arxiv.org/abs/2604.11038 · HTML: https://mlsanity.com/paper/2604.11038/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 75 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 30 of 59)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [RoMa: Robust Dense Feature Matching](https://mlsanity.com/paper/2305.15404/) (2023)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Native and Compact Structured Latents for 3D Generation](https://mlsanity.com/paper/2512.14692/) (2025)
- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation](https://mlsanity.com/paper/2403.09227/) (2024)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects](https://mlsanity.com/paper/2308.07391/) (2023)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [HD-EPIC: A Highly-Detailed Egocentric Video Dataset](https://mlsanity.com/paper/2502.04144/) (2025)
- [Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model](https://mlsanity.com/paper/2410.13882/) (2024)
- [PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos](https://mlsanity.com/paper/2503.17973/) (2025)
- [Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects](https://mlsanity.com/paper/2404.01440/) (2024)
- [Real2Code: Reconstruct Articulated Objects via Code Generation](https://mlsanity.com/paper/2406.08474/) (2024)
- [SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects](https://mlsanity.com/paper/2410.16499/) (2024)
- [ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting](https://mlsanity.com/paper/2502.19459/) (2025)
- [OPD: Single-view 3D Openable Part Detection](https://mlsanity.com/paper/2203.16421/) (2022)
- [Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces](https://mlsanity.com/paper/2503.19199/) (2025)
- [VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning](https://mlsanity.com/paper/2505.12081/) (2025)
- [PhysX-3D: Physical-Grounded 3D Asset Generation](https://mlsanity.com/paper/2507.12465/) (2025)
- [Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics](https://mlsanity.com/paper/2406.10788/) (2024)

## Built on by (most cited first, 1 of 1)

- [FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations](https://mlsanity.com/paper/2609.20817/) (2026)

## Abstract (arXiv)

We present EgoFun3D, a coordinated task formulation, dataset, and benchmark for modeling interactive 3D objects from egocentric videos. Interactive objects are of high interest for embodied AI but scarce, making modeling from readily available real-world videos valuable. Our task focuses on obtaining simulation-ready interactive 3D objects from egocentric video input. While prior work largely focuses on articulations, we capture general cross-part functional mappings (e.g., rotation of stove knob controls stove burner temperature) through function templates, a structured computational representation. Function templates enable precise evaluation and direct compilation into executable code across simulation platforms. To enable comprehensive benchmarking, we introduce a dataset of 517 egocentric videos featuring challenging real-world interactions with paired 3D geometry, segmentation over 2D and 3D, articulation and function template annotations. To tackle the task, we propose a 4-stage pipeline consisting of: 2D part segmentation, reconstruction, articulation estimation, and function template inference. Comprehensive benchmarking shows that the task is challenging for off-the-shelf methods, highlighting avenues for future work.