# Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces

arXiv: https://arxiv.org/abs/2503.19199 · HTML: https://mlsanity.com/paper/2503.19199/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 95 of its 107 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 30 of 95)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)

## Built on by (most cited first, 30 of 54)

- [SuperDec: 3D Scene Decomposition with Superquadric Primitives](https://mlsanity.com/paper/2504.00992/) (2025)
- [Articulated Object Estimation in the Wild](https://mlsanity.com/paper/2509.01708/) (2025)
- [Describe Anything Anywhere At Any Moment](https://mlsanity.com/paper/2512.00565/) (2025)
- [Compile Scene Graphs with Reinforcement Learning](https://mlsanity.com/paper/2504.13617/) (2025)
- [Articulated 3D Scene Graphs for Open-World Mobile Manipulation](https://mlsanity.com/paper/2602.16356/) (2026)
- [KeySG: Hierarchical Keyframe-Based 3D Scene Graphs](https://mlsanity.com/paper/2510.01049/) (2025)
- [SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modeling](https://mlsanity.com/paper/2512.05343/) (2025)
- [MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning](https://mlsanity.com/paper/2512.16909/) (2025)
- [UnLoc: Leveraging Depth Uncertainties for Floorplan Localization](https://mlsanity.com/paper/2509.11301/) (2025)
- [Pandora: Articulated 3D Scene Graphs from Egocentric Vision](https://mlsanity.com/paper/2603.28732/) (2026)
- [Graph World Models: Concepts, Taxonomy, and Future Directions](https://mlsanity.com/paper/2604.27895/) (2026)
- [Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robots](https://mlsanity.com/paper/2509.24966/) (2025)
- [RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.10168/) (2026)
- [FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning](https://mlsanity.com/paper/2604.03696/) (2026)
- [SLAG: Scalable Language-Augmented Gaussian Splatting](https://mlsanity.com/paper/2505.08124/) (2025)
- [Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction](https://mlsanity.com/paper/2510.04714/) (2025)
- [3D Scene Graphs: Open Challenges and Future Directions](https://mlsanity.com/paper/2606.19383/) (2026)
- [Vision Language Models Cannot Plan, but Can They Formalize?](https://mlsanity.com/paper/2509.21576/) (2025)
- [Open-Vocabulary Functional 3D Human-Scene Interaction Generation](https://mlsanity.com/paper/2601.20835/) (2026)
- [MCP4IFC: IFC-Based Building Design Using Large Language Models](https://mlsanity.com/paper/2511.05533/) (2025)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos](https://mlsanity.com/paper/2603.25539/) (2026)
- [T-FunS3D: Task-Driven Hierarchical Open-Vocabulary 3D Functionality Segmentation](https://mlsanity.com/paper/2606.05975/) (2026)
- [Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs](https://mlsanity.com/paper/2606.31471/) (2026)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Action](https://mlsanity.com/paper/2509.19571/) (2025)
- [HouseLayout3D: A Benchmark and Training-Free Baseline for 3D Layout Estimation in the Wild](https://mlsanity.com/paper/2512.02450/) (2025)
- [Memory Over Maps: 3D Object Localization Without Reconstruction](https://mlsanity.com/paper/2603.20530/) (2026)
- [EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates](https://mlsanity.com/paper/2604.11038/) (2026)
- [Analogical Trajectory Transfer](https://mlsanity.com/paper/2605.14393/) (2026)

## Abstract (arXiv)

We introduce the task of predicting functional 3D scene graphs for real-world indoor environments from posed RGB-D images. Unlike traditional 3D scene graphs that focus on spatial relationships of objects, functional 3D scene graphs capture objects, interactive elements, and their functional relationships. Due to the lack of training data, we leverage foundation models, including visual language models (VLMs) and large language models (LLMs), to encode functional knowledge. We evaluate our approach on an extended SceneFun3D dataset and a newly collected dataset, FunGraph3D, both annotated with functional 3D scene graphs. Our method significantly outperforms adapted baselines, including Open3DSG and ConceptGraph, demonstrating its effectiveness in modeling complex scene functionalities. We also demonstrate downstream applications such as 3D question answering and robotic manipulation using functional 3D scene graphs. See our project page at https://openfungraph.github.io