mlsanityData to 7 Oct 2026

Matterport3D: Learning from RGB-D Data in Indoor Environments

arXiv 1709.06158 ↗First posted 18 Sep 2017cs.CVcited by 1,354

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.

Key facts. Cited by 1,354 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 0 of its 48 references that are in the map. First posted on arXiv 18 Sep 2017.

Builds on

0 of 48 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 2; 2019: 2; 2020: 4; 2021: 159; 2022: 188; 2023: 188; 2024: 258; 2025: 267; 2026: 285.

1,354 papers.

Sort:

By area:

  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  3. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  4. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  5. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  7. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  8. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  9. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
  10. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  11. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  12. 2023Learning Interactive Real-World Simulatorscited by 440
38 more
  1. 2022Visual Language Maps for Robot Navigationcited by 362
  2. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  3. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  4. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  5. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  6. 2021Neural RGB-D Surface Reconstructioncited by 312
  7. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  8. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  9. 2021Dense Depth Priors for Neural Radiance Fields from Sparse Input Viewscited by 301
  10. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  11. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  12. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  13. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  14. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  15. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  16. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  17. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  18. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  19. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  20. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  21. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  22. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  23. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  24. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 170
  25. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  26. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  27. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  28. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  29. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  30. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  31. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  32. 2022NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)cited by 156
  33. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
  34. 2022NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimationcited by 151
  35. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  36. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  37. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  38. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136

50 most cited of 1,354 All 1,354 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments
  2. 2020SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans
  3. 2024SceneCraft: Layout-Guided 3D Scene Generation
  4. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  5. 2024Scaled 360 layouts: Revisiting non-central panoramas
  6. 2026Generation of High-Level Concepts in 3D Scene Graphs via Autoregressive Diffusion
  7. 2026Unified 3D Scene Understanding Through Physical World Modeling
  8. 2025ASSIST-3D: Adapted Scene Synthesis for Class-Agnostic 3D Instance Segmentation