mlsanityData to 14 Oct 2026

EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Models

arXiv 2406.10224 ↗First posted 14 Jun 2024cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

The advent of wearable computers enables a new source of context for AI that is embedded in egocentric sensor data. This new egocentric data comes equipped with fine-grained 3D location information and thus presents the opportunity for a novel class of spatial foundation models that are rooted in 3D space. To measure progress on what we term Egocentric Foundation Models (EFMs) we establish EFM3D, a benchmark with two core 3D egocentric perception tasks. EFM3D is the first benchmark for 3D object detection and surface regression on high quality annotated egocentric data of Project Aria. We propose Egocentric Voxel Lifting (EVL), a baseline for 3D EFMs. EVL leverages all available egocentric modalities and inherits foundational capabilities from 2D foundation models. This model, trained on a large simulated dataset, outperforms existing methods on the EFM3D benchmark.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 44 of its 57 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

44 of 57 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Segment Anythingcited by 8,234
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  11. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  12. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
32 more
  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  2. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  3. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  4. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  5. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  6. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 658
  7. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  8. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  9. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  10. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  11. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  12. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  13. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  14. 20163DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructionscited by 336
  15. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  16. 2021NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Videocited by 241
  17. 2020Atlas: End-to-End 3D Scene Reconstruction from Posed Imagescited by 206
  18. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 167
  19. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 160
  20. 2021ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detectioncited by 157
  21. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  22. 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
  23. 2018InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Datasetcited by 87
  24. 2022SimpleRecon: 3D Reconstruction Without 3D Convolutionscited by 83
  25. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  26. 2022Ponder: Point Cloud Pre-training via Neural Renderingcited by 50
  27. 2021ODAM: Object Detection, Association, and Mapping using Posed RGB Videocited by 22
  28. 2020MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videoscited by 18
  29. 2022RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformerscited by 17
  30. 2023Pixel-Aligned Recurrent Queries for Multi-View 3D Object Detectioncited by 14
  31. 2023Temporally Consistent Online Depth Estimation Using Point-Based Fusioncited by 12
  32. 20233D-LFM: Lifting Foundation Modelcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 13.

26 papers.

Sort:

By area:

  1. 2025SAM 3D: 3Dfy Anything in Imagescited by 247
  2. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 110
  3. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 104
  4. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 45
  5. 2026VGGT-Omegacited by 26
  6. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  7. 2026ShapeR: Robust Conditional 3D Shape Generation from Casual Capturescited by 22
  8. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  9. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  10. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
  11. 2026Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3Dcited by 7
  12. 2025SAB3R: Semantic-Augmented Backbone in 3D Reconstructioncited by 6
14 more
  1. 2025Aria Gen 2 Pilot Datasetcited by 6
  2. 2026Pandora: Articulated 3D Scene Graphs from Egocentric Visioncited by 6
  3. 2025Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infillingcited by 4
  4. 2026NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Datacited by 3
  5. 2026EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Taskscited by 3
  6. 2026FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minutecited by 3
  7. 2025Photoreal Scene Reconstruction from an Egocentric Devicecited by 2
  8. 2026Ego-1K -- A Large-Scale Multiview Video Dataset for Egocentric Visioncited by 2
  9. 2026JRM: Joint Reconstruction Model for Multiple Objects without Alignmentcited by 2
  10. 2026Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trendscited by 1
  11. 2026Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputscited by 1
  12. 2025Ev-Layout: A Large-scale Event-based Multi-modal Dataset for Indoor Layout Estimation and Trackingcited by 0
  13. 2026FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMscited by 0
  14. 2026LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homescited by 0