mlsanityData to 14 Oct 2026

Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces

arXiv 2503.19199 ↗First posted 24 Mar 2025cs.CVcited by 54

Not on the map. Why?

Abstract (arXiv)

We introduce the task of predicting functional 3D scene graphs for real-world indoor environments from posed RGB-D images. Unlike traditional 3D scene graphs that focus on spatial relationships of objects, functional 3D scene graphs capture objects, interactive elements, and their functional relationships. Due to the lack of training data, we leverage foundation models, including visual language models (VLMs) and large language models (LLMs), to encode functional knowledge. We evaluate our approach on an extended SceneFun3D dataset and a newly collected dataset, FunGraph3D, both annotated with functional 3D scene graphs. Our method significantly outperforms adapted baselines, including Open3DSG and ConceptGraph, demonstrating its effectiveness in modeling complex scene functionalities. We also demonstrate downstream applications such as 3D question answering and robotic manipulation using functional 3D scene graphs. See our project page at https://openfungraph.github.io

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 95 of its 107 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

95 of 107 references.

References · 95

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
38 more
  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  2. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  3. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  4. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  5. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  6. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  7. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  8. 2023LERF: Language Embedded Radiance Fieldscited by 543
  9. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  10. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  11. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  12. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 385
  13. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  14. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 364
  15. 2016Visual Relationship Detection with Language Priorscited by 334
  16. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  17. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  18. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  19. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  20. 2018Graph R-CNN for Scene Graph Generationcited by 298
  21. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  22. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  23. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  24. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  25. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  26. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 217
  27. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  28. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  29. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  30. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  31. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  32. 2018Point Convolutional Neural Networks by Extension Operatorscited by 154
  33. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  34. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 144
  35. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  36. 20203D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentationcited by 141
  37. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  38. 2018Grounded Human-Object Interaction Hotspots from Videocited by 129

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 37.

54 papers.

Sort:

By area:

  1. 2025SuperDec: 3D Scene Decomposition with Superquadric Primitivescited by 21
  2. 2025Articulated Object Estimation in the Wildcited by 19
  3. 2025Describe Anything Anywhere At Any Momentcited by 19
  4. 2025Compile Scene Graphs with Reinforcement Learningcited by 11
  5. 2026Articulated 3D Scene Graphs for Open-World Mobile Manipulationcited by 11
  6. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 10
  7. 2025SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modelingcited by 9
  8. 2025MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planningcited by 9
  9. 2025UnLoc: Leveraging Depth Uncertainties for Floorplan Localizationcited by 8
  10. 2026Pandora: Articulated 3D Scene Graphs from Egocentric Visioncited by 6
  11. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 6
  12. 2025Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robotscited by 5
38 more
  1. 2026RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generationcited by 5
  2. 2026FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoningcited by 5
  3. 2025SLAG: Scalable Language-Augmented Gaussian Splattingcited by 4
  4. 2025Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Predictioncited by 4
  5. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  6. 2025Vision Language Models Cannot Plan, but Can They Formalize?cited by 3
  7. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generationcited by 3
  8. 2025MCP4IFC: IFC-Based Building Design Using Large Language Modelscited by 2
  9. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  10. 2026PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videoscited by 2
  11. 2026T-FunS3D: Task-Driven Hierarchical Open-Vocabulary 3D Functionality Segmentationcited by 2
  12. 2026Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphscited by 2
  13. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  14. 2025Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Actioncited by 1
  15. 2025HouseLayout3D: A Benchmark and Training-Free Baseline for 3D Layout Estimation in the Wildcited by 1
  16. 2026Memory Over Maps: 3D Object Localization Without Reconstructioncited by 1
  17. 2026EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templatescited by 1
  18. 2026Analogical Trajectory Transfercited by 1
  19. 2026Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spacescited by 1
  20. 2026PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoningcited by 1
  21. 2026OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environmentscited by 1
  22. 2026Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Roboticscited by 1
  23. 2026R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Videocited by 1
  24. 2025BINDER: Instantly Adaptive Mobile Manipulation with Open-Vocabulary Commandscited by 0
  25. 2026UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentationcited by 0
  26. 2026A Survey of Spatial Memory Representations for Efficient Robot Navigationcited by 0
  27. 2026OpenSGA: Efficient 3D Scene Graph Alignment in the Open Worldcited by 0
  28. 2026RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnessescited by 0
  29. 2026From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Modelscited by 0
  30. 2026Scaling Diverse Language Generation for 3D Visual Groundingcited by 0
  31. 2026NoPA: Non-Parametric Online 3D Scene Graph Generationcited by 0
  32. 2026Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environmentscited by 0
  33. 2026GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenescited by 0
  34. 2026Finder: Agentic Closed-Loop Object Finding for Embodied Groundingcited by 0
  35. 2026SnapPhysics: A Physics-Aware Scene Graph from a Single View for Interactive Mixed Reality Scenescited by 0
  36. 2026FunArt: Decoding Functional Structure and Articulation from Generative 3D Latentscited by 0
  37. 2026Geometric and Semantic Coupling for Interaction Understanding in 3D Scenescited by 0
  38. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding
  2. 2026From Pixels to Concepts: Growing Rich 3D Semantic Scene Graph Forests utilizing Foundation Models ↗
  3. 2025ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Models
  4. 2025Open-Vocabulary Indoor Object Grounding with 3D Hierarchical Scene Graph
  5. 2025GeoSceneGraph: Geometric Scene Graph Diffusion Model for Text-guided 3D Indoor Scene Synthesis
  6. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  7. 2026Generation of High-Level Concepts in 3D Scene Graphs via Autoregressive Diffusion
  8. 2025Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning