mlsanityData to 7 Oct 2026

ACPBench: Reasoning about Action, Change, and Planning

arXiv 2410.05669 ↗First posted 8 Oct 2024cs.AIcited by 37

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

There is an increasing body of work using Large Language Models (LLMs) as agents for orchestrating workflows and making decisions in domains that require planning and multi-step reasoning. As a result, it is imperative to evaluate LLMs on core skills required for planning. In this work, we present ACPBench, a benchmark for evaluating the reasoning tasks in the field of planning. The benchmark consists of 7 reasoning tasks over 13 planning domains. The collection is constructed from planning domains described in a formal language. This allows us to synthesize problems with provably correct solutions across many tasks and domains. Further, it allows us the luxury of scale without additional human effort, i.e., many additional problems can be created automatically. Our extensive evaluation of 22 LLMs and OpenAI o1 reasoning models highlights the significant gap in the reasoning capability of the LLMs. Our findings with OpenAI o1, a multi-turn reasoning model, reveal significant gains in performance on multiple-choice questions, yet surprisingly, no notable progress is made on boolean questions. The ACPBench collection is available at https://ibm.github.io/ACPBench.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 19 of its 32 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

19 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023On the Planning Abilities of Large Language Models : A Critical Investigation25 May
  2. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change21 Jun

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  11. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  12. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
5 more
  1. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  2. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  3. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  4. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 93
  5. 2023Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planningcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 15.

37 papers.

Sort:

By area:

  1. 2025Survey on Evaluation of LLM-based Agentscited by 190
  2. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  3. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  4. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  5. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  6. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  7. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 11
  8. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
  9. 2025Seemingly Simple Planning Problems are Computationally Challenging: The Countdown Gamecited by 7
  10. 2025Procedural Knowledge Improves Agentic LLM Workflowscited by 7
  11. 2026On Protecting Agentic Systems' Intellectual Property via Watermarkingcited by 7
  12. 2025How Far Are LLMs from Symbolic Planners? An NLP-Based Perspectivecited by 6
25 more
  1. 2025APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Trainingcited by 6
  2. 2025Frontier Large Language Models Rival State-of-the-Art Plannerscited by 6
  3. 2025ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planningcited by 5
  4. 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scalingcited by 5
  5. 2025Exploring the Hidden Reasoning Process of Large Language Models by Misleading Themcited by 3
  6. 2025Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluationcited by 3
  7. 2025ALAS: Transactional and Dynamic Multi-Agent LLM Planningcited by 3
  8. 2026Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inferencecited by 3
  9. 2025Lossless Vocabulary Reduction for Auto-Regressive Language Modelscited by 2
  10. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  11. 2026Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Taskscited by 2
  12. 2026PACE: A Proxy for Agentic Capability Evaluationcited by 2
  13. 2026ETR: Outcome-Guided Elastic Trust Regions for Policy Optimizationcited by 1
  14. 2026SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluationscited by 1
  15. 2026Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guaranteescited by 1
  16. 2026What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilitiescited by 1
  17. 2026AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worldscited by 1
  18. 2025Make Planning Research Rigorous Again!cited by 0
  19. 2025LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Languagecited by 0
  20. 2025Iterative Deployment Improves Planning Skills in LLMscited by 0
  21. 2026Textual Planning with Explicit Latent Transitionscited by 0
  22. 2026When AI Says It Feelscited by 0
  23. 2026Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedbackcited by 0
  24. 2026Planning with Transformers: Chain of Computation and Structured Context Windowscited by 0
  25. 2026Which LLM is Best for Translating Natural Language Goals to PDDLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planning
  2. 2026SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models
  3. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insights
  4. 2026ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models
  5. 2024ProcBench: Benchmark for Multi-Step Reasoning and Following Procedure
  6. 2025Large Language Models for Planning: A Comprehensive and Systematic Survey
  7. 2024Thinking Forward and Backward: Effective Backward Planning with Large Language Models
  8. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Models