mlsanityData to 7 Oct 2026

Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation

arXiv 2403.17846 ↗First posted 26 Mar 2024cs.RO cs.AI cs.CLcited by 195

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

42 of 55 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Segment Anythingcited by 8,158
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  6. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  7. 2022Language-driven Semantic Segmentationcited by 711
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  9. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  10. 2023LERF: Language Embedded Radiance Fieldscited by 538
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  12. 2022Visual Language Maps for Robot Navigationcited by 362
30 more
  1. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  2. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  3. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  4. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  5. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  6. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  7. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  8. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 175
  9. 2021FAST-LIO2: Fast Direct LiDAR-inertial Odometrycited by 170
  10. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  11. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  12. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  13. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  14. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  15. 2018CubeSLAM: Monocular 3D Object SLAMcited by 101
  16. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  17. 2018Fusion++: Volumetric Object-Level SLAMcited by 97
  18. 2023GOAT: GO to Any Thingcited by 89
  19. 2018QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAMcited by 86
  20. 2019Volumetric Instance-Aware Semantic Mapping and 3D Object Discoverycited by 79
  21. 2024Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulationcited by 69
  22. 2018MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAMcited by 65
  23. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 55
  24. 2023SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environmentscited by 53
  25. 2023Collaborative Dynamic 3D Scene Graphs for Automated Drivingcited by 34
  26. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  27. 2023GRID: Scene-Graph-based Instruction-driven Robotic Task Planningcited by 27
  28. 2022Situational Graphs for Robot Navigation in Structured Indoor Environmentscited by 21
  29. 2023Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planningcited by 16
  30. 2022Leveraging Large (Visual) Language Models for Robot 3D Scene Understandingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 82; 2026: 93.

195 papers.

Sort:

By area:

  1. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  2. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 51
  3. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  4. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  5. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  6. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  7. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  8. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  9. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  10. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  11. 2025RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Explorationcited by 23
  12. 2025FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interactioncited by 22
38 more
  1. 2025Describe Anything Anywhere At Any Momentcited by 19
  2. 2025Articulated Object Estimation in the Wildcited by 18
  3. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 16
  4. 2024RelationField: Relate Anything in Radiance Fieldscited by 16
  5. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 14
  6. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  7. 2024Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Modelscited by 12
  8. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 12
  9. 2024Open-Vocabulary Online Semantic Mapping for SLAMcited by 11
  10. 2024The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphscited by 11
  11. 2025Universal Scene Graph Generationcited by 11
  12. 2025Air-Ground Collaboration for Language-Specified Missions in Unknown Environmentscited by 11
  13. 2025Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videoscited by 10
  14. 2025RAVEN: Resilient Aerial Navigation via Open-Set Semantic Memory and Behavior Adaptationcited by 10
  15. 2025CHOrD: Generation of Collision-Free, House-Scale, and Organized Digital Twins for 3D Indoor Scenes with Controllable Floor Plans and Optimal Layoutscited by 9
  16. 2025MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoningcited by 9
  17. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  18. 2026Articulated 3D Scene Graphs for Open-World Mobile Manipulationcited by 9
  19. 2025GaussianGraph: 3D Gaussian-based Scene Graph Generation for Open-world Scene Understandingcited by 8
  20. 2025IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenescited by 8
  21. 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysiscited by 8
  22. 2025FindAnything: Open-Vocabulary and Object-Centric Mapping for Robot Exploration in Any Environmentcited by 8
  23. 2025SG-Reg: Generalizable and Efficient Scene Graph Registrationcited by 8
  24. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 8
  25. 2025MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planningcited by 8
  26. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
  27. 2024CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversationscited by 7
  28. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  29. 2025Unified Semantic Transformer for 3D Scene Understandingcited by 7
  30. 2025DualMap: Online Open-Vocabulary Semantic Mapping for Natural Language Navigation in Dynamic Changing Scenescited by 6
  31. 2025Multimodal Spatial Language Maps for Robot Navigation and Manipulationcited by 6
  32. 2025Language-Grounded Hierarchical Planning and Execution with Multi-Robot 3D Scene Graphscited by 6
  33. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 6
  34. 2025ObjectReact: Learning Object-Relative Control for Visual Navigationcited by 5
  35. 2025Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robotscited by 5
  36. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  37. 2025RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Modelscited by 5
  38. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5

50 most cited of 195 All 195 (JSON)