mlsanityData to 14 Oct 2026

Habitat-Matterport 3D Semantics Dataset

arXiv 2210.05633 ↗First posted 11 Oct 2022cs.CVcited by 110

Not on the map. Why?

Abstract (arXiv)

We present the Habitat-Matterport 3D Semantics (HM3DSEM) dataset. HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics that is currently available to the academic community. It consists of 142,646 object instance annotations across 216 3D spaces and 3,100 rooms within those spaces. The scale, quality, and diversity of object annotations far exceed those of prior datasets. A key difference setting apart HM3DSEM from other datasets is the use of texture information to annotate pixel-accurate object boundaries. We demonstrate the effectiveness of HM3DSEM dataset for the Object Goal Navigation task using different methods. Policies trained using HM3DSEM perform outperform those trained on prior datasets. Introduction of HM3DSEM in the Habitat ObjectNav Challenge lead to an increase in participation from 400 submissions in 2021 to 1022 submissions in 2022.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

34 of 41 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  6. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  8. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  9. 2018On Evaluation of Embodied Navigation Agentscited by 547
  10. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  11. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  12. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
22 more
  1. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  2. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  3. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  4. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  6. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  7. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  8. 2020Neural Topological SLAM for Visual Navigationcited by 195
  9. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 193
  10. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  11. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  12. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  13. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 129
  14. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  15. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  16. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 66
  17. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 65
  18. 2021Learning to Map for Active Semantic Goal Navigationcited by 50
  19. 2021SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistencycited by 48
  20. 2020Semantic Curiosity for Active Visual Learningcited by 45
  21. 2018CHALET: Cornell House Agent Learning Environmentcited by 38
  22. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 18; 2024: 31; 2025: 29; 2026: 28.

110 papers.

Sort:

By area:

  1. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  2. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  3. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 167
  4. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  5. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
  6. 2022Navigating to Objects in the Real Worldcited by 106
  7. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 94
  8. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 93
  9. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  10. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 70
  11. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 67
  12. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 67
38 more
  1. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 65
  2. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 61
  3. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  4. 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 55
  5. 2023Navigating to Objects Specified by Imagescited by 51
  6. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  7. 2024GaussNav: Gaussian Splatting for Visual Navigationcited by 49
  8. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 49
  9. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  10. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  11. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  12. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  13. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  14. 2024Prioritized Semantic Learning for Zero-shot Instance Navigationcited by 31
  15. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  16. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  17. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 23
  18. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  19. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  20. 2023Harmonic Mobile Manipulationcited by 17
  21. 2023Object Goal Navigation with Recursive Implicit Mapscited by 16
  22. 2024Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigationcited by 15
  23. 2024TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigationcited by 15
  24. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  25. 2024BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulationcited by 14
  26. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  27. 2023MOPA: Modular Object Navigation with PointGoal Agentscited by 11
  28. 2024Personalized Instance-based Navigation Toward User-Specific Objects in Realistic Environmentscited by 11
  29. 20233D-LLM: Injecting the 3D World into Large Language Modelscited by 10
  30. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 10
  31. 2023HabiCrowd: A High Performance Simulator for Crowd-Aware Visual Navigationcited by 9
  32. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  33. 2024Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Descriptioncited by 9
  34. 2023Multimodal Large Language Model for Visual Navigationcited by 8
  35. 2025FindAnything: Open-Vocabulary and Object-Centric Mapping for Robot Exploration in Any Environmentcited by 8
  36. 2024Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Visioncited by 7
  37. 2023E(2)-Equivariant Graph Planning for Navigationcited by 6
  38. 2025DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigationcited by 6

50 most cited of 110 All 110 (JSON)