mlsanityData to 7 Oct 2026

PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change

arXiv 2206.10498 ↗First posted 21 Jun 2022cs.CL cs.AIcited by 372

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Generating plans of action, and reasoning about change have long been considered a core competence of intelligent agents. It is thus no surprise that evaluating the planning and reasoning capabilities of large language models (LLMs) has become a hot topic of research. Most claims about LLM planning capabilities are however based on common sense tasks-where it becomes hard to tell whether LLMs are planning or merely retrieving from their vast world knowledge. There is a strong need for systematic and extensible planning benchmarks with sufficient diversity to evaluate whether LLMs have innate planning capabilities. Motivated by this, we propose PlanBench, an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change. PlanBench provides sufficient diversity in both the task domains and the specific planning capabilities. Our studies also show that on many critical capabilities-including plan generation-LLM performance falls quite short, even with the SOTA models. PlanBench can thus function as a useful marker of progress of LLMs in planning and reasoning.

Key facts. Cited by 372 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 17 of its 35 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

17 of 35 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
5 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 35; 2024: 107; 2025: 124; 2026: 104.

372 papers.

Sort:

By area:

  1. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  2. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  3. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  5. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  6. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  7. 2025Survey on Evaluation of LLM-based Agentscited by 190
  8. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  9. 2022Language models show human-like content effects on reasoning taskscited by 171
  10. 2024The pitfalls of next-token predictioncited by 156
  11. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  12. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
38 more
  1. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  2. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  3. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  4. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
  5. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 93
  6. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  7. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  8. 2024Video as the New Language for Real-World Decision Makingcited by 86
  9. 2024Can Large Language Models Reason and Plan?cited by 82
  10. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  11. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  12. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  13. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 76
  14. 2024Can large language models explore in-context?cited by 75
  15. 2024Building Machines that Learn and Think with Peoplecited by 73
  16. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  17. 2023Strategic Reasoning with Language Modelscited by 64
  18. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  19. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 62
  20. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  21. 2025LLM Agents for Education: Advances and Applicationscited by 59
  22. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  23. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  24. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  25. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  26. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  27. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  28. 2023The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactionscited by 45
  29. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 43
  30. 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 42
  31. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  32. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
  33. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  34. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  35. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  36. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  37. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  38. 2023Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Modelscited by 37

50 most cited of 372 All 372 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
  2. 2023On the Planning Abilities of Large Language Models : A Critical Investigation
  3. 2026GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI
  4. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  5. 2025LLMs Can Plan Only If We Tell Them
  6. 2025Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents
  7. 2024Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1
  8. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoning