mlsanityData to 7 Oct 2026

PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities

arXiv 2504.14773 ↗First posted 21 Apr 2025cs.AI cs.CL cs.LGcited by 6

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Planning is central to agents and agentic AI. The ability to plan, e.g., creating travel itineraries within a budget, holds immense potential in both scientific and commercial contexts. Moreover, optimal plans tend to require fewer resources compared to ad-hoc methods. To date, a comprehensive understanding of existing planning benchmarks appears to be lacking. Without it, comparing planning algorithms' performance across domains or selecting suitable algorithms for new scenarios remains challenging. In this paper, we examine a range of planning benchmarks to identify commonly used testbeds for algorithm development and highlight potential gaps. These benchmarks are categorized into embodied environments, web navigation, scheduling, games and puzzles, and everyday task automation. Our study recommends the most appropriate benchmarks for various algorithms and offers insights to guide future benchmark development.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 49 of its 63 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

49 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions4 Mar
  2. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agents6 Feb
  3. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  4. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments4 Nov
  5. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation8 Feb
  6. 2024Understanding the planning of LLM agents: A survey5 Feb
  7. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan
  9. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct

Other references · 40

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 411
28 more
  1. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  3. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  4. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  5. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  6. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  7. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  8. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  9. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  10. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  11. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  12. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 81
  13. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arenacited by 78
  14. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Taskscited by 78
  15. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  16. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  17. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoningcited by 49
  18. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  19. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  20. 2024Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Tracescited by 40
  21. 2024Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programmingcited by 35
  22. 2023ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Gamescited by 22
  23. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMscited by 21
  24. 2021What do Large Language Models Learn about Scripts?cited by 19
  25. 2023TaskLAMA: Probing the Complex Task Understanding of Language Modelscited by 12
  26. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activitiescited by 9
  27. 2024Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios?cited by 7
  28. 2024WorldAPIs: The World Is Worth How Many APIs? A Thought Experimentcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worldscited by 6
  2. 2026Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agentscited by 6
  3. 2025Cost-Awareness in Tree-Search LLM Planning: A Systematic Studycited by 5
  4. 2026TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agentscited by 4
  5. 2025Benchmark for Planning and Control with Large Language Model Agents: Blocksworld with Model Context Protocolcited by 3
  6. 2026LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Seemingly Simple Planning Problems are Computationally Challenging: The Countdown Game
  2. 2025Beyond Itinerary Planning-A Real-World Benchmark for Multi-Turn and Tool-Using Travel Tasks
  3. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
  4. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints
  5. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agents
  6. 2026MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
  7. 2025Large Language Models for Planning: A Comprehensive and Systematic Survey
  8. 2025REALM-Bench: A Benchmark for Evaluating Multi-Agent Systems on Real-world, Dynamic Planning and Scheduling Tasks