mlsanityData to 7 Oct 2026

Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs

arXiv 2309.15940 ↗First posted 27 Sep 2023cs.RO cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

We present an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework for grounding a variety of entities, such as object instances, agents, and regions, with free-form text-based queries. Unlike conventional semantic-based object localization approaches, our system facilitates context-aware entity localization, allowing for queries such as ``pick up a cup on a kitchen table" or ``navigate to a sofa on which someone is sitting". In contrast to existing research on 3D scene graphs, OVSG supports free-form text input and open-vocabulary querying. Through a series of comparative experiments using the ScanNet dataset and a self-collected dataset, we demonstrate that our proposed approach significantly surpasses the performance of previous semantic-based localization techniques. Moreover, we highlight the practical application of OVSG in real-world robot navigation and manipulation experiments.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

28 of 40 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2017Mask R-CNNcited by 7,259
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2016Modeling Context in Referring Expressionscited by 1,054
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  10. 2020ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAMcited by 862
  11. 2022Language-driven Semantic Segmentationcited by 711
  12. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
16 more
  1. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  2. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  3. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  4. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  5. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  6. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  7. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  8. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  9. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  10. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  11. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  12. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  13. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  14. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  15. 2022Scene-level Tracking and Reconstruction without Object Priorscited by 4
  16. 2023Mono-STAR: Mono-camera Scene-level Tracking and Reconstructioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 10; 2026: 8.

33 papers.

Sort:

By area:

  1. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  2. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  3. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  4. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  5. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  6. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  7. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  8. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33
  9. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  10. 2024OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environmentscited by 22
  11. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  12. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
21 more
  1. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 14
  2. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Mapscited by 13
  3. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  4. 2025OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Mergingcited by 12
  5. 2024OpenSU3D: Open World 3D Scene Understanding using Foundation Modelscited by 6
  6. 2024QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understandingcited by 5
  7. 2025CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graphcited by 4
  8. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  9. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  10. 2024Time is on my sight: scene graph filtering for dynamic environment perception in an LLM-driven robotcited by 2
  11. 2025OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditionscited by 2
  12. 2025L3M+P: Lifelong Planning with Large Language Modelscited by 2
  13. 2026T-FunS3D: Task-Driven Hierarchical Open-Vocabulary 3D Functionality Segmentationcited by 2
  14. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  15. 2024Guessing human intentions to avoid dangerous situations in caregiving robotscited by 0
  16. 2025DSM: Constructing a Diverse Semantic Map for 3D Visual Groundingcited by 0
  17. 2026Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphscited by 0
  18. 2026CAPruner: Conceptual-Adjacent Scene Graph Pruner for Enhancing 3D Spatial Reasoning of Large Language Modelscited by 0
  19. 2026OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environmentscited by 0
  20. 2026Holo-Captioning: Toward the Text Equivalent of 3D Scenescited by 0
  21. 2026TDVR: Joint Text Disambiguation and Viewpoint Reasoning for Zero-Shot 3D Visual Groundingcited by 0