mlsanityData to 13 Oct 2026

Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships

arXiv 2402.12259 ↗First posted 19 Feb 2024cs.CVcited by 76

Not on the map. Why?

Abstract (arXiv)

Current approaches for 3D scene graph prediction rely on labeled datasets to train models for a fixed set of known object classes and relationship categories. We present Open3DSG, an alternative approach to learn 3D scene graph prediction in an open world without requiring labeled scene graph data. We co-embed the features from a 3D scene graph prediction backbone with the feature space of powerful open world 2D vision language foundation models. This enables us to predict 3D scene graphs from 3D point clouds in a zero-shot manner by querying object classes from an open vocabulary and predicting the inter-object relationships from a grounded LLM with scene graph features and queried object classes as context. Open3DSG is the first 3D point cloud method to predict not only explicit open-vocabulary object classes, but also open-set relationships that are not limited to a predefined label set, making it possible to express rare as well as specific objects and relationships in the predicted 3D scene graph. Our experiments show that Open3DSG is effective at predicting arbitrary object classes as well as their complex inter-object relationships describing spatial, supportive, semantic and comparative relationships.

Key facts. Cited by 76 papers in the map (data through 13 Oct 2026). Builds on 54 of its 61 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

54 of 61 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,231
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,096
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,168
  11. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 606
  12. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 557
38 more
  1. 2023LERF: Language Embedded Radiance Fieldscited by 538
  2. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 525
  3. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  4. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  5. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  6. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 359
  7. 2016Visual Relationship Detection with Language Priorscited by 333
  8. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  9. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 300
  10. 2018Graph R-CNN for Scene Graph Generationcited by 296
  11. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 292
  12. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  13. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  14. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 239
  15. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  16. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  17. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 178
  18. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 176
  19. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  20. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 160
  21. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  22. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 142
  23. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  24. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  25. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  26. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 83
  27. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  28. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  29. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  30. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  31. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  32. 2021Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysiscited by 47
  33. 2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 42
  34. 2023Incremental 3D Semantic Scene Graph Prediction from RGB Sequencescited by 39
  35. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  36. 2023Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphscited by 33
  37. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  38. 2023SGAligner : 3D Scene Alignment with Scene Graphscited by 30

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 12; 2025: 24; 2026: 40.

76 papers.

Sort:

By area:

  1. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  2. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  3. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  4. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  5. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  6. 2025FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interactioncited by 22
  7. 2025SuperDec: 3D Scene Decomposition with Superquadric Primitivescited by 20
  8. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  9. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 17
  10. 2024RelationField: Relate Anything in Radiance Fieldscited by 16
  11. 2025OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Mergingcited by 12
  12. 2024The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphscited by 11
38 more
  1. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 9
  2. 2026Articulated 3D Scene Graphs for Open-World Mobile Manipulationcited by 9
  3. 2025Unified Semantic Transformer for 3D Scene Understandingcited by 8
  4. 2025MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planningcited by 8
  5. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  6. 2026ReLaGS: Relational Language Gaussian Splattingcited by 6
  7. 2024Vocabulary-Free 3D Instance Segmentation with Vision and Language Assistantcited by 5
  8. 2025Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robotscited by 5
  9. 2026RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generationcited by 5
  10. 2026FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoningcited by 5
  11. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 5
  12. 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
  13. 2025Learning 3D Scene Analogies with Neural Contextual Scene Mapscited by 4
  14. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  15. 2024Generation of Uncertainty-Aware High-Level Spatial Concepts in Factorized 3D Scene Graphs via Graph Neural Networkscited by 3
  16. 2024TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understandingcited by 3
  17. 2025OpenCity3D: What do Vision-Language Models know about Urban Environments?cited by 3
  18. 2025BoxFusion: Reconstruction-Free Open-Vocabulary 3D Object Detection via Real-Time Multi-View Box Fusioncited by 3
  19. 2025ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understandingcited by 3
  20. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  21. 2025RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregationcited by 2
  22. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understandingcited by 2
  23. 2025L3M+P: Lifelong Planning with Large Language Modelscited by 2
  24. 2025Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Predictioncited by 2
  25. 2026LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environmentscited by 2
  26. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  27. 2026T-FunS3D: Task-Driven Hierarchical Open-Vocabulary 3D Functionality Segmentationcited by 2
  28. 2026Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphscited by 2
  29. 2025AIvaluateXR: An Evaluation Framework for on-Device AI in XR with Benchmarking Resultscited by 1
  30. 2025OpenLex3D: A Tiered Evaluation Benchmark for Open-Vocabulary 3D Scene Representationscited by 1
  31. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  32. 2025Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Actioncited by 1
  33. 2026VIZOR: Viewpoint-Invariant Zero-Shot Scene Graph Generation for 3D Scene Reasoningcited by 1
  34. 2026Analogical Trajectory Transfercited by 1
  35. 2026Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spacescited by 1
  36. 2026PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoningcited by 1
  37. 2026OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environmentscited by 1
  38. 2026Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Roboticscited by 1

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis
  2. 2024OpenSU3D: Open World 3D Scene Understanding using Foundation Models
  3. 2026Open-Set 3D Scene Graphs for Field Robotics: An Outdoor Case Study ↗
  4. 2024Open-Vocabulary Octree-Graph for 3D Scene Understanding
  5. 2025ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Models
  6. 2026DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors
  7. 2025Open World Scene Graph Generation using Vision Language Models
  8. 2025Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning