mlsanityData to 7 Oct 2026

Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning

arXiv 2305.07716 ↗First posted 12 May 2023cs.RO cs.AIcited by 16

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Long-horizon task planning is essential for the development of intelligent assistive and service robots. In this work, we investigate the applicability of a smaller class of large language models (LLMs), specifically GPT-2, in robotic task planning by learning to decompose tasks into subgoal specifications for a planner to execute sequentially. Our method grounds the input of the LLM on the domain that is represented as a scene graph, enabling it to translate human requests into executable robot plans, thereby learning to reason over long-horizon tasks, as encountered in the ALFRED benchmark. We compare our approach with classical planning and baseline methods to examine the applicability and generalizability of LLM-based planners. Our findings suggest that the knowledge stored in an LLM can be effectively grounded to perform long-horizon task planning, demonstrating the promising potential for the future application of neuro-symbolic planning methods in robotics.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 58 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

23 of 58 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  12. 2022Visual Language Maps for Robot Navigationcited by 362
11 more
  1. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  2. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  3. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  4. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  5. 2022The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMscited by 57
  6. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  7. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  8. 2022CAPE: Corrective Actions from Precondition Errors using Large Language Modelscited by 30
  9. 2021Extended Tree Search for Robot Task and Motion Planningcited by 14
  10. 2022Accelerating Integrated Task and Motion Planning with Neural Feasibility Checkingcited by 10
  11. 2022Leveraging Large (Visual) Language Models for Robot 3D Scene Understandingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 4; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  3. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  4. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  5. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  6. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  7. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  8. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  9. 2024Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languagescited by 32
  10. 2023LLM-Based Human-Robot Collaboration Framework for Manipulation Taskscited by 17
  11. 2023Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lackingcited by 7
  12. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
4 more
  1. 2025Large Language Models as Common-Sense Heuristicscited by 2
  2. 2025Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaborationcited by 1
  3. 2026Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Searchcited by 0
  4. 2026Hybrid LLM-based Intelligent Framework for Robot Task Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  2. 2024Fast and Accurate Task Planning using Neuro-Symbolic Language Models and Multi-level Goal Decomposition
  3. 2026GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning
  4. 2025A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
  5. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  6. 2025Combining Large Language Models and Symbolic Reasoning for Multi-Robot Temporal Planning through Explainable Knowledge Bases
  7. 2024Can only LLMs do Reasoning?: Potential of Small Language Models in Task Planning
  8. 2024Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback