mlsanityData to 7 Oct 2026

Look Further Ahead: Testing the Limits of GPT-4 in Path Planning

arXiv 2406.12000 ↗First posted 17 Jun 2024cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown impressive capabilities across a wide variety of tasks. However, they still face challenges with long-horizon planning. To study this, we propose path planning tasks as a platform to evaluate LLMs' ability to navigate long trajectories under geometric constraints. Our proposed benchmark systematically tests path-planning skills in complex settings. Using this, we examined GPT-4's planning abilities using various task representations and prompting approaches. We found that framing prompts as Python code and decomposing long trajectory tasks improve GPT-4's path planning effectiveness. However, while these approaches show some promise toward improving the planning ability of the model, they do not obtain optimal paths and fail at generalizing over extended horizons.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 43 of its 47 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

43 of 47 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
31 more
  1. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  2. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  4. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  5. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  6. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  7. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  8. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  9. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  10. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  11. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  12. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  13. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  14. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  15. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  16. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  17. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  18. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  19. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  20. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  21. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  22. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arenacited by 78
  23. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 54
  24. 2023Understanding the Capabilities of Large Language Models for Automated Planningcited by 34
  25. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  26. 2023LLM A*: Human in the Loop Large Language Models Enabled A* Search for Roboticscited by 26
  27. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  28. 2024Why Solving Multi-agent Path Finding with Large Language Model has not Succeeded Yetcited by 22
  29. 2024Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMscited by 19
  30. 2021Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?cited by 15
  31. 2023On the Planning, Search, and Memorization Capabilities of Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  2. 2025A Survey on Large Language Models for Automated Planningcited by 17
  3. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
  4. 2025Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoningcited by 2
  5. 2025LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?cited by 2
  6. 2026MMP-A*: Multimodal Perception Enhanced Incremental Heuristic Search on Path Planningcited by 0
  7. 2026Test-Driven Agentic Framework for Reliable Robot Controllercited by 0