mlsanityData to 14 Oct 2026

LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench

arXiv 2409.13373 ↗First posted 20 Sep 2024cs.AI cs.CLcited by 94

Not on the map. Why?

Cited by 43 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities. PlanBench, an extensible benchmark we developed in 2022, soon after the release of GPT3, has remained an important tool for evaluating the planning abilities of LLMs. Despite the slew of new private and open source LLMs since GPT3, progress on this benchmark has been surprisingly slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs--making it a new kind of model: a Large Reasoning Model (LRM). Using this development as a catalyst, this paper takes a comprehensive look at how well current LLMs and new LRMs do on PlanBench. As we shall see, while o1's performance is a quantum improvement on the benchmark, outpacing the competition, it is still far from saturating it. This improvement also brings to the fore questions about accuracy, efficiency, and guarantees which must be considered before deploying such systems.

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). Builds on 10 of its 49 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

10 of 49 references.

References · 10

  1. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  2. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  3. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  4. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  5. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 179
  6. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 128
  7. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
  8. 2024Can Large Language Models Reason and Plan?cited by 82
  9. 2024Thought of Search: Planning with Language Models Through The Lens of Efficiencycited by 32
  10. 2021GPT3-to-plan: Extracting plans from text using GPT-3cited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 56; 2026: 20.

94 papers.

Sort:

By area:

  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  2. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  4. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 92
  5. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  6. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 77
  7. 2024ACPBench: Reasoning about Action, Change, and Planningcited by 39
  8. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  9. 2024Benchmarking Agentic Workflow Generationcited by 37
  10. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 34
  11. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 34
  12. 2025InSTA: Towards Internet-Scale Training For Agentscited by 32
38 more
  1. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30
  2. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  3. 2025Frontier LLMs Still Struggle with Simple Reasoning Taskscited by 21
  4. 2024Revealing the Barriers of Language Agents in Planningcited by 20
  5. 2024On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizabilitycited by 18
  6. 2024On the Limit of Language Models as Planning Formalizerscited by 18
  7. 2025Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Trackingcited by 18
  8. 2025A Survey on Large Language Models for Automated Planningcited by 17
  9. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  10. 2025Evaluating the Goal-Directedness of Large Language Modelscited by 16
  11. 2025PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solvingcited by 15
  12. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  13. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  14. 2025Generating Symbolic World Models via Test-time Scaling of Large Language Modelscited by 12
  15. 2025Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloringcited by 10
  16. 2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectoriescited by 10
  17. 2024WILT: A Multi-Turn, Memorization-Robust Inductive Logic Benchmark for LLMscited by 9
  18. 2024Robust Planning with Compound LLM Architectures: An LLM-Modulo Approachcited by 9
  19. 2024Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approachcited by 9
  20. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Studycited by 8
  21. 2024TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzlescited by 7
  22. 2025ChatHTN: Interleaving Approximate (LLM) and Symbolic HTN Planningcited by 7
  23. 2025Language Models For Generalised PDDL Planning: Synthesising Sound and Programmatic Policiescited by 7
  24. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
  25. 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarkscited by 6
  26. 2025Complex LLM Planning via Automated Heuristics Discoverycited by 6
  27. 2025Reasoning Large Language Model Errors Arise from Hallucinating Critical Problem Featurescited by 6
  28. 2024Trust but Verify: Programmatic VLM Evaluation in the Wildcited by 5
  29. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 5
  30. 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scalingcited by 5
  31. 2025Electronic Circuit Principles of Large Language Modelscited by 4
  32. 2025The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Modelscited by 4
  33. 2025Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scalingcited by 4
  34. 2025Reasoning Models Reason Well, Until They Don'tcited by 4
  35. 2025CoS: Towards Optimal Event Scheduling via Chain-of-Schedulingcited by 4
  36. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  37. 2025Large Language and Reasoning Models are Shallow Disjunctive Reasonerscited by 3
  38. 2025TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Gamescited by 3

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1
  2. 2025LLMs Can Plan Only If We Tell Them
  3. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
  4. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example
  5. 2023Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planning
  6. 2024Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
  7. 2025Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
  8. 2025Thinking Machines: A Survey of LLM based Reasoning Strategies