mlsanityData to 7 Oct 2026

SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation

arXiv 2410.08189 ↗First posted 10 Oct 2024cs.CV cs.ROcited by 106

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a new framework for zero-shot object navigation. Existing zero-shot object navigation methods prompt LLM with the text of spatially closed objects, which lacks enough scene context for in-depth reasoning. To better preserve the information of environment and fully exploit the reasoning ability of LLM, we propose to represent the observed scene with 3D scene graph. The scene graph encodes the relationships between objects, groups and rooms with a LLM-friendly structure, for which we design a hierarchical chain-of-thought prompt to help LLM reason the goal location according to scene context by traversing the nodes and edges. Moreover, benefit from the scene graph representation, we further design a re-perception mechanism to empower the object navigation framework with the ability to correct perception error. We conduct extensive experiments on MP3D, HM3D and RoboTHOR environments, where SG-Nav surpasses previous state-of-the-art zero-shot methods by more than 10% SR on all benchmarks, while the decision process is explainable. To the best of our knowledge, SG-Nav is the first zero-shot method that achieves even higher performance than supervised object navigation methods on the challenging MP3D benchmark.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 47 of its 50 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

47 of 50 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
35 more
  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  3. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  4. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  6. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  7. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  8. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  9. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  10. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  11. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  12. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  13. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  14. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  15. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  16. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  17. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  18. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  19. 2018Visual Semantic Navigation using Scene Priorscited by 127
  20. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  21. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  22. 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
  23. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  24. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  25. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  26. 2023Embodied Task Planning with Large Language Modelscited by 84
  27. 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 73
  28. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 66
  29. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  30. 2020Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentationcited by 40
  31. 2024EmbodiedSAM: Online Segment Any 3D Thing in Real Timecited by 40
  32. 2021Hierarchical Object-to-Zone Graph for Object Navigationcited by 34
  33. 2020Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agentscited by 27
  34. 2024Memory-based Adapters for Online 3D Scene Perceptioncited by 13
  35. 2024Embodied Instruction Following in Unknown Environmentscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 39; 2026: 64.

106 papers.

Sort:

By area:

  1. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  2. 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
  3. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  4. 2025CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Spacecited by 31
  5. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  6. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  7. 2025Describe Anything Anywhere At Any Momentcited by 19
  8. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  9. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  10. 2025Compile Scene Graphs with Reinforcement Learningcited by 11
  11. 2025IGL-Nav: Incremental 3D Gaussian Localization for Image-goal Navigationcited by 10
  12. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
38 more
  1. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  2. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  3. 2025LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Studycited by 9
  4. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memorycited by 8
  5. 2025DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigationcited by 6
  6. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 6
  7. 2024ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind Peoplecited by 5
  8. 2025SayCoNav: Utilizing Large Language Models for Adaptive Collaboration in Decentralized Multi-Robot Navigationcited by 5
  9. 2025ObjectReact: Learning Object-Relative Control for Visual Navigationcited by 5
  10. 2025Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robotscited by 5
  11. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  12. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  13. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  14. 2025SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuningcited by 5
  15. 2026RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generationcited by 5
  16. 2026LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigationcited by 5
  17. 2026GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoningcited by 5
  18. 2026FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoningcited by 5
  19. 2025VL-Nav: Neuro-Symbolic Reasoning-based Vision-Language Navigationcited by 4
  20. 2025Long-Term Planning Around Humans in Domestic Environments with 3D Scene Graphscited by 4
  21. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  22. 2025RATE-Nav: Region-Aware Termination Enhancement for Zero-shot Object Navigation with Vision-Language Modelscited by 4
  23. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  24. 2025BoxFusion: Reconstruction-Free Open-Vocabulary 3D Object Detection via Real-Time Multi-View Box Fusioncited by 3
  25. 2025Distilling LLM Prior to Flow Model for Generalizable Agent's Imagination in Object Goal Navigationcited by 3
  26. 2025Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigationcited by 3
  27. 2025SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimizationcited by 3
  28. 2026Explore Like Humans: Autonomous Exploration with Online SG-Memo Construction for Embodied Agentscited by 3
  29. 2026Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigationcited by 3
  30. 2025LGR: LLM-Guided Ranking of Frontiers for Object Goal Navigationcited by 2
  31. 2025RANa: Retrieval-Augmented Navigationcited by 2
  32. 2025From Data to Modeling: Fully Open-vocabulary Scene Graph Generationcited by 2
  33. 2025SGN-CIRL: Scene Graph-based Navigation with Curriculum, Imitation, and Reinforcement Learningcited by 2
  34. 2025Constructing coherent spatial memory in LLM agents through graph rectificationcited by 2
  35. 2025Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Predictioncited by 2
  36. 2026LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environmentscited by 2
  37. 2026ReMemNav: Memory-Based Decision Correction and Target Verification for Zero-Shot Object Navigationcited by 2
  38. 2026SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answeringcited by 2

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs
  2. 2026Exploring Bottlenecks in VLM-LLM Navigation: How 3D Scene Understanding Capability Impacts Zero-Shot VLN
  3. 20263DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splatting
  4. 2024TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigation
  5. 2026HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models
  6. 2026SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation
  7. 2025BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation
  8. 2026SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment