# Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning

arXiv: https://arxiv.org/abs/2511.05894 · HTML: https://mlsanity.com/paper/2511.05894/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 37 references that are in the map.
- First posted on arXiv 8 Nov 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 8 of 8)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models](https://mlsanity.com/paper/2207.11514/) (2022)
- [Adapter is All You Need for Tuning Visual Tasks](https://mlsanity.com/paper/2311.15010/) (2023)
- [EntityCLIP: Entity-Centric Image-Text Matching via Multimodal Attentive Contrastive Learning](https://mlsanity.com/paper/2410.17810/) (2024)

## Built on by (most cited first, 6 of 6)

- [Graph World Models: Concepts, Taxonomy, and Future Directions](https://mlsanity.com/paper/2604.27895/) (2026)
- [Hypothesis Graph Refinement: Hypothesis-Driven Exploration with Cascade Error Correction for Embodied Navigation](https://mlsanity.com/paper/2604.04108/) (2026)
- [Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memory](https://mlsanity.com/paper/2607.11529/) (2026)
- [SGR3 Model: Scene Graph Retrieval-Reasoning Model in 3D](https://mlsanity.com/paper/2603.04614/) (2026)
- [GraphWrit3R: End-to-End 3D Scene Graph Writing](https://mlsanity.com/paper/2609.31595/) (2026)
- [GHOST: Geometry-Constrained Hybrid Diffusion for Open-Vocabulary 3D Scene Graph Generation](https://mlsanity.com/paper/2610.16455/) (2026)

## Abstract (arXiv)

Understanding 3D scenes in open-world settings poses fundamental challenges for vision and robotics, particularly due to the limitations of closed-vocabulary supervision and static annotations. To address this, we propose a unified framework for Open-World 3D Scene Graph Generation with Retrieval-Augmented Reasoning, which enables generalizable and interactive 3D scene understanding. Our method integrates Vision-Language Models (VLMs) with retrieval-based reasoning to support multimodal exploration and language-guided interaction. The framework comprises two key components: (1) a dynamic scene graph generation module that detects objects and infers semantic relationships without fixed label sets, and (2) a retrieval-augmented reasoning pipeline that encodes scene graphs into a vector database to support text/image-conditioned queries. We evaluate our method on 3DSSG and Replica benchmarks across four tasks-scene question answering, visual grounding, instance retrieval, and task planning-demonstrating robust generalization and superior performance in diverse environments. Our results highlight the effectiveness of combining open-vocabulary perception with retrieval-based reasoning for scalable 3D scene understanding.