# SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation

arXiv: https://arxiv.org/abs/2410.08189 · HTML: https://mlsanity.com/paper/2410.08189/

## Key facts

- Cited by 109 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 50 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2312.03275/) (2023)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings](https://mlsanity.com/paper/2206.12403/) (2022)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning](https://mlsanity.com/paper/2201.10029/) (2022)

## Built on by (most cited first, 30 of 109)

- [UniGoal: Towards Universal Zero-shot Goal-oriented Navigation](https://mlsanity.com/paper/2503.10630/) (2025)
- [MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation](https://mlsanity.com/paper/2503.13446/) (2025)
- [CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMs](https://mlsanity.com/paper/2412.10439/) (2024)
- [CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space](https://mlsanity.com/paper/2502.12532/) (2025)
- [Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities](https://mlsanity.com/paper/2506.18019/) (2025)
- [Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods](https://mlsanity.com/paper/2511.15722/) (2025)
- [g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks](https://mlsanity.com/paper/2411.17030/) (2024)
- [Describe Anything Anywhere At Any Moment](https://mlsanity.com/paper/2512.00565/) (2025)
- [MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation](https://mlsanity.com/paper/2511.10376/) (2025)
- [Compile Scene Graphs with Reinforcement Learning](https://mlsanity.com/paper/2504.13617/) (2025)
- [IGL-Nav: Incremental 3D Gaussian Localization for Image-goal Navigation](https://mlsanity.com/paper/2508.00823/) (2025)
- [Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation System](https://mlsanity.com/paper/2606.18112/) (2026)
- [GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigation](https://mlsanity.com/paper/2509.10454/) (2025)
- [VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memory](https://mlsanity.com/paper/2601.08665/) (2026)
- [LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study](https://mlsanity.com/paper/2505.19510/) (2025)
- [PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memory](https://mlsanity.com/paper/2511.06840/) (2025)
- [GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving](https://mlsanity.com/paper/2511.11266/) (2025)
- [DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation](https://mlsanity.com/paper/2505.21969/) (2025)
- [ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People](https://mlsanity.com/paper/2412.03118/) (2024)
- [SayCoNav: Utilizing Large Language Models for Adaptive Collaboration in Decentralized Multi-Robot Navigation](https://mlsanity.com/paper/2505.13729/) (2025)
- [ObjectReact: Learning Object-Relative Control for Visual Navigation](https://mlsanity.com/paper/2509.09594/) (2025)
- [Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robots](https://mlsanity.com/paper/2509.24966/) (2025)
- [EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval](https://mlsanity.com/paper/2510.18546/) (2025)
- [IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation](https://mlsanity.com/paper/2511.17384/) (2025)
- [Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration](https://mlsanity.com/paper/2512.02458/) (2025)
- [SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2512.14757/) (2025)
- [RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.10168/) (2026)
- [LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigation](https://mlsanity.com/paper/2602.12351/) (2026)
- [GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning](https://mlsanity.com/paper/2603.19137/) (2026)
- [FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning](https://mlsanity.com/paper/2604.03696/) (2026)

## Abstract (arXiv)

In this paper, we propose a new framework for zero-shot object navigation. Existing zero-shot object navigation methods prompt LLM with the text of spatially closed objects, which lacks enough scene context for in-depth reasoning. To better preserve the information of environment and fully exploit the reasoning ability of LLM, we propose to represent the observed scene with 3D scene graph. The scene graph encodes the relationships between objects, groups and rooms with a LLM-friendly structure, for which we design a hierarchical chain-of-thought prompt to help LLM reason the goal location according to scene context by traversing the nodes and edges. Moreover, benefit from the scene graph representation, we further design a re-perception mechanism to empower the object navigation framework with the ability to correct perception error. We conduct extensive experiments on MP3D, HM3D and RoboTHOR environments, where SG-Nav surpasses previous state-of-the-art zero-shot methods by more than 10% SR on all benchmarks, while the decision process is explainable. To the best of our knowledge, SG-Nav is the first zero-shot method that achieves even higher performance than supervised object navigation methods on the challenging MP3D benchmark.