mlsanityData to 7 Oct 2026

On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)

arXiv 2302.06706 ↗First posted 13 Feb 2023cs.AI cs.CL cs.LGcited by 85

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) how good LLMs are by themselves in generating and validating simple plans in commonsense planning tasks (of the type that humans are generally quite good at) and (2) how good LLMs are in being a source of heuristic guidance for other agents--either AI planners or human planners--in their planning tasks. To investigate these questions in a systematic rather than anecdotal manner, we start by developing a benchmark suite based on the kinds of domains employed in the International Planning Competition. On this benchmark, we evaluate LLMs in three modes: autonomous, heuristic and human-in-the-loop. Our results show that LLM's ability to autonomously generate executable plans is quite meager, averaging only about 3% success rate. The heuristic and human-in-the-loop modes show slightly more promise. In addition to these results, we also make our benchmark and evaluation tools available to support investigations by research community.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 25 of its 51 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

25 of 51 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
13 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2022A Generalist Agentcited by 802
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  11. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  12. 2022On the Paradox of Learning to Reason from Datacited by 93
  13. 2018Discovering Underlying Plans Based on Shallow Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 19; 2025: 27; 2026: 9.

85 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  4. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  5. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  6. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  7. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  8. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  9. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  10. 2023Large Language Models can Learn Rulescited by 66
  11. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  12. 2023The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactionscited by 45
38 more
  1. 2025LLM Agents Making Agent Toolscited by 45
  2. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  3. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  4. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  5. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  6. 2024Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Tracescited by 40
  7. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  8. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  9. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  10. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  11. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  12. 2023Understanding the Capabilities of Large Language Models for Automated Planningcited by 34
  13. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 30
  14. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  15. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  16. 2023Gentopia: A Collaborative Platform for Tool-Augmented LLMscited by 25
  17. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  18. 2025Agents of Change: Self-Evolving LLM Agents for Strategic Planningcited by 20
  19. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  20. 2024On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizabilitycited by 18
  21. 2025Classical Planning with LLM-Generated Heuristics: Challenging the State of the Art with Python Codecited by 18
  22. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 18
  23. 2024On the Limit of Language Models as Planning Formalizerscited by 17
  24. 2023Exploiting Language Models as a Source of Knowledge for Cognitive Agentscited by 15
  25. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  26. 2023REX: Rapid Exploration and eXploitation for AI Agentscited by 13
  27. 2024How Can LLM Guide RL? A Value-Based Approachcited by 13
  28. 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Modelscited by 12
  29. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Modelscited by 12
  30. 2023ProTIP: Progressive Tool Retrieval Improves Planningcited by 12
  31. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  32. 2025SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Modelscited by 10
  33. 2025AIA Forecaster: Technical Reportcited by 9
  34. 2023Embodied Active Learning of Relational State Abstractions for Bilevel Planningcited by 8
  35. 2025TripTide: A Benchmark for Adaptive Travel Planning under Disruptionscited by 8
  36. 2023Egocentric Planning for Scalable Embodied Task Achievementcited by 7
  37. 2024Look Further Ahead: Testing the Limits of GPT-4 in Path Planningcited by 7
  38. 2024Transforming Agency. On the mode of existence of Large Language Modelscited by 7

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Planning Abilities of Large Language Models : A Critical Investigation
  2. 2025LLMs Can Plan Only If We Tell Them
  3. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
  4. 2024Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
  5. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Models
  6. 2024Understanding the planning of LLM agents: A survey
  7. 2024Revealing the Barriers of Language Agents in Planning
  8. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study