# Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1

arXiv: https://arxiv.org/abs/2410.02162 · HTML: https://mlsanity.com/paper/2410.02162/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024)
- [On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks](https://mlsanity.com/paper/2402.08115/) (2024)
- [NATURAL PLAN: Benchmarking LLMs on Natural Language Planning](https://mlsanity.com/paper/2406.04520/) (2024)
- [Chain of Thoughtlessness? An Analysis of CoT in Planning](https://mlsanity.com/paper/2405.04776/) (2024)
- [Limitations of Language Models in Arithmetic and Symbolic Induction](https://mlsanity.com/paper/2208.05051/) (2022)
- [Data Contamination Through the Lens of Time](https://mlsanity.com/paper/2310.10628/) (2023)
- [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023)
- [GPT3-to-plan: Extracting plans from text using GPT-3](https://mlsanity.com/paper/2106.07131/) (2021)

## Built on by (most cited first, 17 of 17)

- [LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models](https://mlsanity.com/paper/2503.18971/) (2025)
- [Classical Planning with LLM-Generated Heuristics: Challenging the State of the Art with Python Code](https://mlsanity.com/paper/2503.18809/) (2025)
- [HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinking](https://mlsanity.com/paper/2505.02322/) (2025)
- [ChatHTN: Interleaving Approximate (LLM) and Symbolic HTN Planning](https://mlsanity.com/paper/2505.11814/) (2025)
- [Procedural Knowledge Improves Agentic LLM Workflows](https://mlsanity.com/paper/2511.07568/) (2025)
- [Frontier Large Language Models Rival State-of-the-Art Planners](https://mlsanity.com/paper/2511.09378/) (2025)
- [HARBOR: Exploring Persona Dynamics in Multi-Agent Competition](https://mlsanity.com/paper/2502.12149/) (2025)
- [Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions](https://mlsanity.com/paper/2503.02238/) (2025)
- [TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games](https://mlsanity.com/paper/2506.10209/) (2025)
- [Rethinking the Unsolvable: When In-Context Search Meets Test-Time Scaling](https://mlsanity.com/paper/2505.22290/) (2025)
- [Baba is LLM: Reasoning in a Game with Dynamic Rules](https://mlsanity.com/paper/2506.19095/) (2025)
- [Visualizing Thought: Conceptual Diagrams Enable Robust Planning in LMMs](https://mlsanity.com/paper/2503.11790/) (2025)
- [Using Large Language Models for Abstraction of Planning Domains - Extended Version](https://mlsanity.com/paper/2510.20258/) (2025)
- [World-Model Collapse as a Phase Transition](https://mlsanity.com/paper/2606.31399/) (2026)
- [Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors](https://mlsanity.com/paper/2505.23323/) (2025)
- [Bootstrapping Human-Like Planning via LLMs](https://mlsanity.com/paper/2506.22604/) (2025)
- [LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Language](https://mlsanity.com/paper/2510.05972/) (2025)

## Abstract (arXiv)

The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities, but -- despite the slew of new private and open source LLMs since GPT3 -- progress has remained slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs -- making it a new kind of model: a Large Reasoning Model (LRM). In this paper, we evaluate the planning capabilities of two LRMs (o1-preview and o1-mini) on both planning and scheduling benchmarks. We see that while o1 does seem to offer significant improvements over autoregressive LLMs, this comes at a steep inference cost, while still failing to provide any guarantees over what it generates. We also show that combining o1 models with external verifiers -- in a so-called LRM-Modulo system -- guarantees the correctness of the combined system's output while further improving performance.