mlsanityData to 14 Oct 2026

On the Planning Abilities of Large Language Models : A Critical Investigation

arXiv 2305.15771 ↗First posted 25 May 2023cs.AIcited by 312

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) the effectiveness of LLMs in generating plans autonomously in commonsense planning tasks and (2) the potential of LLMs in LLM-Modulo settings where they act as a source of heuristic guidance for external planners and verifiers. We conduct a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic. Our findings reveal that LLMs' ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of ~12% across the domains. However, the results in the LLM-Modulo setting show more promise. In the LLM-Modulo setting, we demonstrate that LLM-generated plans can improve the search process for underlying sound planners and additionally show that external verifiers can help provide feedback on the generated plans and back-prompt the LLM for better plan generation.

Key facts. Cited by 312 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 17 of its 40 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

17 of 40 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  12. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
5 more
  1. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  2. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  3. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  4. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  5. 2022On the Paradox of Learning to Reason from Datacited by 93

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 120; 2025: 92; 2026: 88.

312 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  4. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  5. 2024Many-Shot In-Context Learningcited by 206
  6. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 179
  7. 2024The pitfalls of next-token predictioncited by 160
  8. 2024Tree Search for Language Model Agentscited by 151
  9. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  11. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 142
  12. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
38 more
  1. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 128
  2. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  3. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
  4. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  5. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
  6. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  7. 2025Efficient Reasoning Models: A Surveycited by 95
  8. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 94
  9. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 92
  10. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  11. 2024Can Large Language Models Reason and Plan?cited by 82
  12. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  13. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  14. 2026AutoHarness: improving LLM agents by automatically synthesizing a code harnesscited by 76
  15. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 68
  16. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 64
  17. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  18. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  19. 2024Can Language Models Serve as Text-Based World Simulators?cited by 52
  20. 2024Beyond Preferences in AI Alignmentcited by 49
  21. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  22. 2023Large Language Models are Zero Shot Hypothesis Proposerscited by 45
  23. 2023CoPAL: Corrective Planning of Robot Actions with Large Language Modelscited by 40
  24. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 39
  25. 2024ACPBench: Reasoning about Action, Change, and Planningcited by 39
  26. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 39
  27. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 37
  28. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 37
  29. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  30. 2024Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programmingcited by 35
  31. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  32. 2024Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluationcited by 34
  33. 2024Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languagescited by 33
  34. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  35. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  36. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  37. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  38. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30

50 most cited of 312 All 312 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
  2. 2025LLMs Can Plan Only If We Tell Them
  3. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
  4. 2024Understanding the planning of LLM agents: A survey
  5. 2025Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
  6. 2025Large Language Models for Planning: A Comprehensive and Systematic Survey
  7. 2024Query-Efficient Planning with Language Models
  8. 2025Language Model as Planner and Formalizer under Constraints