# The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphs

arXiv: https://arxiv.org/abs/2412.01539 · HTML: https://mlsanity.com/paper/2412.01539/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [SAI3D: Segment Any Instance in 3D Scenes](https://mlsanity.com/paper/2312.11557/) (2023)
- [Clio: Real-time Task-Driven Open-Set 3D Scene Graphs](https://mlsanity.com/paper/2404.13696/) (2024)
- [Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships](https://mlsanity.com/paper/2402.12259/) (2024)
- [Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systems](https://mlsanity.com/paper/2305.07154/) (2023)
- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [FutureMapping: The Computational Structure of Spatial AI Systems](https://mlsanity.com/paper/1803.11288/) (2018)
- [Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping](https://mlsanity.com/paper/1909.04631/) (2019)
- [Language-EXtended Indoor SLAM (LEXIS): A Versatile System for Real-time Visual Scene Understanding](https://mlsanity.com/paper/2309.15065/) (2023)
- [LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories](https://mlsanity.com/paper/2311.12174/) (2023)
- [MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognition](https://mlsanity.com/paper/2311.18402/) (2023)
- [Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations](https://mlsanity.com/paper/2311.17938/) (2023)
- [Deep Learning-Based 3D Instance and Semantic Segmentation: A Review](https://mlsanity.com/paper/2406.13308/) (2024)

## Built on by (most cited first, 11 of 11)

- [FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interaction](https://mlsanity.com/paper/2503.07909/) (2025)
- [Describe Anything Anywhere At Any Moment](https://mlsanity.com/paper/2512.00565/) (2025)
- [GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving](https://mlsanity.com/paper/2511.11266/) (2025)
- [Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting](https://mlsanity.com/paper/2503.05949/) (2025)
- [Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction](https://mlsanity.com/paper/2510.04714/) (2025)
- [QueryAdapter: Rapid Adaptation of Vision-Language Models in Response to Natural Language Queries](https://mlsanity.com/paper/2502.18735/) (2025)
- [OpenLex3D: A Tiered Evaluation Benchmark for Open-Vocabulary 3D Scene Representations](https://mlsanity.com/paper/2503.19764/) (2025)
- [LIEREx: Language-Image Embeddings for Robotic Exploration](https://mlsanity.com/paper/2602.01930/) (2026)
- [DISC: Dense Integrated Semantic Context for Large-Scale Open-Set Semantic Mapping](https://mlsanity.com/paper/2603.03935/) (2026)
- [Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments](https://mlsanity.com/paper/2608.06170/) (2026)
- [TRACKGRAPH: Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking](https://mlsanity.com/paper/2609.31005/) (2026)

## Abstract (arXiv)

3D open-vocabulary scene graph methods are a promising map representation for embodied agents, however many current approaches are computationally expensive. In this paper, we reexamine the critical design choices established in previous works to optimize both efficiency and performance. We propose a general scene graph framework and conduct three studies that focus on image pre-processing, feature fusion, and feature selection. Our findings reveal that commonly used image pre-processing techniques provide minimal performance improvement while tripling computation (on a per object view basis). We also show that averaging feature labels across different views significantly degrades performance. We study alternative feature selection strategies that enhance performance without adding unnecessary computational costs. Based on our findings, we introduce a computationally balanced approach for 3D point cloud segmentation with per-object features. The approach matches state-of-the-art classification accuracy while achieving a threefold reduction in computation.