mlsanityData to 7 Oct 2026

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

arXiv 2307.06135 ↗First posted 12 Jul 2023cs.RO cs.AIcited by 289

Not on the map. Why?

Cited by 63 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics. We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a 'semantic search' for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an 'iterative replanning' pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and natural language instruction for a mobile manipulator robot to execute. We provide real robot video demonstrations on our project page https://sayplan.github.io.

Key facts. Cited by 289 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

34 of 57 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
22 more
  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  2. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  5. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  6. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  7. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  8. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  9. 2023Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPTcited by 78
  10. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  11. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  12. 2021Semantics for Robotic Mapping, Perception and Interaction: A Surveycited by 56
  13. 2021Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networkscited by 50
  14. 2023ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Applicationcited by 48
  15. 2023Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Promptingcited by 45
  16. 2020Planning with Learned Object Importance in Large Problem Instances using Graph Neural Networkscited by 38
  17. 2018Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoningcited by 27
  18. 2022Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Roboticscited by 23
  19. 2020CAMPs: Learning Context-Specific Abstractions for Efficient Planning in Factored MDPscited by 18
  20. 2021A Holistic Approach to Reactive Mobile Manipulationcited by 15
  21. 2023Contrastive Language, Action, and State Pre-training for Robot Learningcited by 15
  22. 2020Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Searchcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 94; 2025: 92; 2026: 87.

289 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  4. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  5. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  6. 2023Video Understanding with Large Language Models: A Surveycited by 203
  7. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  8. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  9. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  10. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  11. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  12. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
38 more
  1. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  2. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  3. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  4. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  5. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  6. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  7. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  8. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  9. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  10. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  11. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  12. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  13. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  14. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  15. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  16. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  17. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  18. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  19. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  20. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  21. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  22. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  23. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  24. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  25. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  26. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  27. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  28. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  29. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  30. 2023CoPAL: Corrective Planning of Robot Actions with Large Language Modelscited by 39
  31. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39
  32. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  33. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  34. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  35. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  36. 2023Interactive Task Planning with Language Modelscited by 36
  37. 2024Exploration of LLM Multi-Agent Application Implementation Based on LangGraph+CrewAIcited by 35
  38. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33

50 most cited of 289 All 289 (JSON)