# On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability

arXiv: https://arxiv.org/abs/2409.19924 · HTML: https://mlsanity.com/paper/2409.19924/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench](https://mlsanity.com/paper/2409.13373/) (2024)
- [On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)](https://mlsanity.com/paper/2302.06706/) (2023)
- [Memory3: Language Modeling with Explicit Memory](https://mlsanity.com/paper/2407.01178/) (2024)
- [MM3DGS SLAM: Multi-modal 3D Gaussian Splatting for SLAM Using Vision, Depth, and Inertial Measurements](https://mlsanity.com/paper/2404.00923/) (2024)
- [Fine-Tuning Language Models Using Formal Methods Feedback](https://mlsanity.com/paper/2310.18239/) (2023)
- [Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search](https://mlsanity.com/paper/2212.14849/) (2022)

## Built on by (most cited first, 18 of 18)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?](https://mlsanity.com/paper/2503.12349/) (2025)
- [ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning](https://mlsanity.com/paper/2502.16268/) (2025)
- [Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework](https://mlsanity.com/paper/2411.01639/) (2024)
- [Generating Symbolic World Models via Test-time Scaling of Large Language Models](https://mlsanity.com/paper/2502.04728/) (2025)
- [Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark](https://mlsanity.com/paper/2512.05091/) (2025)
- [Learning from Streaming Video with Orthogonal Gradients](https://mlsanity.com/paper/2504.01961/) (2025)
- [VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigation](https://mlsanity.com/paper/2509.18592/) (2025)
- [Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impact](https://mlsanity.com/paper/2507.00951/) (2025)
- [LR^2Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems](https://mlsanity.com/paper/2502.17848/) (2025)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [Weaker LLMs' Opinions Also Matter: Mixture of Opinions Enhances LLM's Mathematical Reasoning](https://mlsanity.com/paper/2502.19622/) (2025)
- [LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs](https://mlsanity.com/paper/2506.10527/) (2025)
- [Text Chunking for Document Classification for Urban System Management using Large Language Models](https://mlsanity.com/paper/2504.00274/) (2025)
- [Sensory-Motor Control with Large Language Models via Iterative Policy Refinement](https://mlsanity.com/paper/2506.04867/) (2025)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have showcased their ability to perform complex reasoning tasks, but their effectiveness in planning remains underexplored. In this study, we evaluate the planning capabilities of OpenAI's o1 models across a variety of benchmark tasks, focusing on three key aspects: feasibility, optimality, and generalizability. Through empirical evaluations on constraint-heavy tasks (e.g., textitBarman, textitTyreworld) and spatially complex environments (e.g., textitTermes, textitFloortile), we highlight o1-preview's strengths in self-evaluation and constraint-following, while also identifying bottlenecks in decision-making and memory management, particularly in tasks requiring robust spatial reasoning. Our results reveal that o1-preview outperforms GPT-4 in adhering to task constraints and managing state transitions in structured environments. However, the model often generates suboptimal solutions with redundant actions and struggles to generalize effectively in spatially complex tasks. This pilot study provides foundational insights into the planning limitations of LLMs, offering key directions for future research on improving memory management, decision-making, and generalization in LLM-based planning. Code available at https://github.com/VITA-Group/o1-planning.