mlsanityData to 7 Oct 2026

EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI

arXiv 2312.16170 ↗First posted 26 Dec 2023cs.CV cs.AI cs.ROcited by 146

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the realm of computer vision and robotics, embodied agents are expected to explore their environment and carry out human instructions. This necessitates the ability to fully understand 3D scenes given their first-person observations and contextualize them into language for interaction. However, traditional research focuses more on scene-level input and output setups from a global view. To address the gap, we introduce EmbodiedScan, a multi-modal, ego-centric 3D perception dataset and benchmark for holistic 3D scene understanding. It encompasses over 5k scans encapsulating 1M ego-centric RGB-D views, 1M language prompts, 160k 3D-oriented boxes spanning over 760 categories, some of which partially align with LVIS, and dense semantic occupancy with 80 common categories. Building upon this database, we introduce a baseline framework named Embodied Perceptron. It is capable of processing an arbitrary number of multi-modal inputs and demonstrates remarkable 3D perception capabilities, both within the two series of benchmarks we set up, i.e., fundamental 3D perception tasks and language-grounded tasks, and in the wild. Codes, datasets, and benchmarks will be available at https://github.com/OpenRobotLab/EmbodiedScan.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Builds on 57 of its 67 references that are in the map. First posted on arXiv 26 Dec 2023.

Builds on

57 of 67 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Segment Anythingcited by 8,158
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  11. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
38 more
  1. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  2. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  4. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  5. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  6. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  7. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  8. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  9. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  11. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  12. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  13. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  14. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  15. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  16. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  17. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  18. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  19. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  20. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  21. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  22. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  23. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  24. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  25. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  26. 2021Group-Free 3D Object Detection via Transformerscited by 215
  27. 2023Scene as Occupancycited by 208
  28. 2020SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimationcited by 202
  29. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  30. 2021Probabilistic and Geometric Depth: Detecting Objects in Perspectivecited by 190
  31. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  32. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  33. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  34. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
  35. 2021ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detectioncited by 157
  36. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  37. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  38. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 69; 2026: 48.

146 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  3. 2025Streaming 4D Visual Geometry Transformercited by 147
  4. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  5. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  8. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  9. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  10. 2025Visual Spatial Tuningcited by 85
  11. 2024Diffusion Models in Low-Level Vision: A Surveycited by 73
  12. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
38 more
  1. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 68
  2. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  3. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  4. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  5. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  6. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  7. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  8. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  9. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  10. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 43
  11. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  12. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  13. 2025VolSplat: Rethinking Feed-Forward 3D Gaussian Splatting with Voxel-Aligned Predictioncited by 41
  14. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  15. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  16. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  17. 2025Detect Anything 3D in the Wildcited by 32
  18. 20253D Aware Region Prompted Vision Language Modelcited by 31
  19. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  20. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  21. 2025UAVScenes: A Multi-Modal Dataset for UAVscited by 26
  22. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  23. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understandingcited by 24
  24. 2025Embodied AI: From LLMs to World Modelscited by 23
  25. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  26. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  27. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentationcited by 20
  28. 2025InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layoutscited by 18
  29. 2024Language Driven Occupancy Predictioncited by 17
  30. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 17
  31. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  32. 2024DAOcc: 3D Object Detection Assisted Multi-Sensor Fusion for 3D Occupancy Predictioncited by 15
  33. 2024Panoptic-FlashOcc: An Efficient Baseline to Marry Semantic Occupancy with Panoptic via Instance Centercited by 14
  34. 2025UniForward: Unified 3D Scene and Semantic Field Reconstruction via Feed-Forward Gaussian Splatting from Only Sparse-View Imagescited by 13
  35. 2025DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Groundingcited by 12
  36. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  37. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  38. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11

50 most cited of 146 All 146 (JSON)