mlsanityData to 7 Oct 2026

Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI

arXiv 2109.08238 ↗First posted 16 Sep 2021cs.CV cs.AIcited by 579

Not on the map. Why?

Cited by 60 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the Habitat-Matterport 3D (HM3D) dataset. HM3D is a large-scale dataset of 1,000 building-scale 3D reconstructions from a diverse set of real-world locations. Each scene in the dataset consists of a textured 3D mesh reconstruction of interiors such as multi-floor residences, stores, and other private indoor spaces. HM3D surpasses existing datasets available for academic research in terms of physical scale, completeness of the reconstruction, and visual fidelity. HM3D contains 112.5k m^2 of navigable space, which is 1.4 - 3.7x larger than other building-scale datasets such as MP3D and Gibson. When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w.r.t. counterpart images captured with real cameras, and HM3D meshes have 34 - 91% fewer artifacts due to incomplete surface reconstruction. The increased scale, fidelity, and diversity of HM3D directly impacts the performance of embodied AI agents trained using it. In fact, we find that HM3D is `pareto optimal' in the following sense -- agents trained to perform PointGoal navigation on HM3D achieve the highest performance regardless of whether they are evaluated on HM3D, Gibson, or MP3D. No similar claim can be made about training on other datasets. HM3D-trained PointNav agents achieve 100% performance on Gibson-test dataset, suggesting that it might be time to retire that episode dataset.

Key facts. Cited by 579 papers in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

32 of 52 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  5. 2018Demystifying MMD GANscited by 1,268
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  9. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2018On Evaluation of Embodied Navigation Agentscited by 526
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
20 more
  1. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  2. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  3. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  4. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  5. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  6. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  7. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  8. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  9. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  10. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  11. 2018Semi-parametric Topological Memory for Navigationcited by 203
  12. 2020Neural Topological SLAM for Visual Navigationcited by 193
  13. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  14. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  15. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 104
  16. 2019CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysiscited by 49
  17. 2018CHALET: Cornell House Agent Learning Environmentcited by 38
  18. 2021Differentiable SLAM-net: Learning Particle SLAM for Visual Navigationcited by 34
  19. 2020Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learningcited by 27
  20. 2019Rescan: Inductive Instance Segmentation for Indoor RGBD Scanscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 37; 2023: 73; 2024: 120; 2025: 162; 2026: 184.

579 papers.

Sort:

By area:

  1. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  3. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  4. 2023Learning Interactive Real-World Simulatorscited by 440
  5. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  6. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  7. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  8. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  9. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  10. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  11. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  12. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
38 more
  1. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  2. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  5. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 170
  6. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  7. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  8. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  9. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  10. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  11. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  12. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 136
  13. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  14. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  15. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  16. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  17. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  18. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 108
  19. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  20. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  21. 2022Navigating to Objects in the Real Worldcited by 104
  22. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  23. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  24. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  25. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  26. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  27. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  28. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  29. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  30. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  31. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  32. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  33. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  34. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  35. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  36. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 66
  37. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  38. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63

50 most cited of 579 All 579 (JSON)