mlsanityData to 7 Oct 2026

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

arXiv 2309.16650 ↗First posted 28 Sep 2023cs.RO cs.CVcited by 357

Not on the map. Why?

Cited by 48 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )

Key facts. Cited by 357 papers in the map (data through 7 Oct 2026). Builds on 57 of its 89 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

57 of 89 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  10. 2022Language-driven Semantic Segmentationcited by 711
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
38 more
  1. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 586
  2. 2021Extract Free Dense Labels from CLIPcited by 580
  3. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  4. 2023LERF: Language Embedded Radiance Fieldscited by 538
  5. 2021Image Segmentation Using Text and Image Promptscited by 509
  6. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  7. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  9. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  10. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  11. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  12. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  13. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  14. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  15. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  16. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  17. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  18. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  19. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 175
  20. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  21. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  22. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  23. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  24. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  25. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 134
  26. 2018Fusion++: Volumetric Object-Level SLAMcited by 97
  27. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 97
  28. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  29. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  30. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  31. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  32. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  33. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  34. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  35. 2022Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understandingcited by 37
  36. 2023Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphscited by 33
  37. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  38. 2018Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoningcited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 76; 2025: 115; 2026: 159.

357 papers.

Sort:

By area:

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  5. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  6. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  7. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  8. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  9. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  10. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  11. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  12. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
38 more
  1. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  2. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  3. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  4. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  5. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  6. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  7. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  8. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  9. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  10. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 46
  11. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  12. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  13. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  14. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  15. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  16. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  17. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  18. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  19. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  20. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  21. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  22. 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
  23. 20243D Vision-Language Gaussian Splattingcited by 30
  24. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  25. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 30
  26. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  27. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  28. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  29. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  30. 2024RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigationcited by 25
  31. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  32. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  33. 2025Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentationcited by 23
  34. 2025RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Explorationcited by 23
  35. 2024OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environmentscited by 22
  36. 2025FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interactioncited by 22
  37. 2025ReferSplat: Referring Segmentation in 3D Gaussian Splattingcited by 20
  38. 2024Revisit Anything: Visual Place Recognition via Image Segment Retrievalcited by 19

50 most cited of 357 All 357 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Pixels to Concepts: Growing Rich 3D Semantic Scene Graph Forests utilizing Foundation Models ↗
  2. 2026A Scene Graph Backed Approach to Open Set Semantic Mapping
  3. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding
  4. 2026Open-Set 3D Scene Graphs for Field Robotics: An Outdoor Case Study ↗
  5. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
  6. 2026Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning
  7. 2026Relational Scene Graphs for Object Grounding of Natural Language Commands
  8. 2024Open-Vocabulary Octree-Graph for 3D Scene Understanding