mlsanityData to 7 Oct 2026

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

arXiv 2606.04874 ↗First posted 3 Jun 2026cs.CLcited by 6

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200 τ2-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution benchmarks. The APB benchmark and code are available in hrefhttps://github.com/Mikivishy/AgentPlanningBenchmarkthis URL.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 19 of its 47 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

19 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities21 Apr
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges27 Mar
  3. 2024Plancraft: an evaluation dataset for planning with LLM agents30 Dec
  4. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  5. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security10 Jan

Other references · 14

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2025Qwen3-VL Technical Reportcited by 2,534
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  8. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  9. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  10. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  11. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  12. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 93
2 more
  1. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  2. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agentscited by 2
  2. 2026A2E : An End-to-End Agent Auditing Enginecited by 1
  3. 2026AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worldscited by 1
  4. 2026Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harnesscited by 0
  5. 2026FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Managementcited by 0
  6. 2026Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Executioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
  2. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agents
  3. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
  4. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
  5. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs
  6. 2025Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
  7. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents
  8. 2026Consistent Plan-Act for Long-Horizon Agentic Tasks