mlsanityData to 7 Oct 2026

NATURAL PLAN: Benchmarking LLMs on Natural Language Planning

arXiv 2406.04520 ↗First posted 6 Jun 2024cs.CL cs.AIcited by 125

Not on the map. Why?

Cited by 49 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce NATURAL PLAN, a realistic planning benchmark in natural language containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar Scheduling. We focus our evaluation on the planning capabilities of LLMs with full information on the task, by providing outputs from tools such as Google Flights, Google Maps, and Google Calendar as contexts to the models. This eliminates the need for a tool-use environment for evaluating LLMs on Planning. We observe that NATURAL PLAN is a challenging benchmark for state of the art models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only achieve 31.1% and 34.8% solve rate respectively. We find that model performance drops drastically as the complexity of the problem increases: all models perform below 5% when there are 10 cities, highlighting a significant gap in planning in natural language for SoTA LLMs. We also conduct extensive ablation studies on NATURAL PLAN to further shed light on the (in)effectiveness of approaches such as self-correction, few-shot generalization, and in-context planning with long-contexts on improving LLM planning.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

17 of 34 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  9. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  10. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  11. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  12. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
5 more
  1. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  2. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  3. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  4. 2018Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systemscited by 40
  5. 2020Task-Motion Planning for Safe and Efficient Urban Drivingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 70; 2026: 32.

125 papers.

Sort:

By area:

  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2025Survey on Evaluation of LLM-based Agentscited by 190
  3. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 93
  6. 2025Efficient Reasoning Models: A Surveycited by 93
  7. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  8. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  9. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  10. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 73
  11. 2025Self-rewarding correction for mathematical reasoningcited by 67
  12. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 65
38 more
  1. 2025Evolving Deeper LLM Thinkingcited by 58
  2. 2025SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scalingcited by 54
  3. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  4. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  5. 2024Benchmarking Agentic Workflow Generationcited by 36
  6. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  7. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
  8. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  9. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30
  10. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  11. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  12. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  13. 2025RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agentscited by 24
  14. 2024To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planningcited by 21
  15. 2025Self-Evolving Multi-Agent Systems via Textual Backpropagationcited by 21
  16. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Modelscited by 19
  17. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  18. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 19
  19. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  20. 2024TRIP-PAL: Travel Planning with Guarantees by Combining Large Language Models and Automated Plannerscited by 17
  21. 2024Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1cited by 17
  22. 2025SwarmAgentic: Towards Fully Automated Agentic System Generation via Swarm Intelligencecited by 17
  23. 2024RePrompt: Planning by Automatic Prompt Engineering for Large Language Models Agentscited by 15
  24. 2025PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solvingcited by 15
  25. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  26. 2025Lost at the Beginning of Reasoningcited by 15
  27. 2025SAMULE: Self-Learning Agents Enhanced by Multi-level Reflectioncited by 15
  28. 2025HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinkingcited by 14
  29. 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Studycited by 12
  30. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  31. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 11
  32. 2024CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Planscited by 10
  33. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  34. 2025Understanding and Steering the Cognitive Behaviors of Reasoning Models at Test-Timecited by 10
  35. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
  36. 2025Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agentscited by 9
  37. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 9
  38. 2024Robust Planning with Compound LLM Architectures: An LLM-Modulo Approachcited by 8

50 most cited of 125 All 125 (JSON)