mlsanityData to 7 Oct 2026

Reasoning with Language Model is Planning with World Model

arXiv 2305.14992 ↗First posted 24 May 2023cs.CL cs.AI cs.LGcited by 846

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have shown remarkable reasoning capabilities, especially when prompted to generate intermediate reasoning steps (e.g., Chain-of-Thought, CoT). However, LLMs can still struggle with problems that are easy for humans, such as generating action plans for executing tasks in a given environment, or performing complex math, logical, and commonsense reasoning. The deficiency stems from the key fact that LLMs lack an internal textitworld model to predict the world textitstate (e.g., environment status, intermediate variable values) and simulate long-term outcomes of actions. This prevents LLMs from performing deliberate planning akin to human brains, which involves exploring alternative reasoning paths, anticipating future states and rewards, and iteratively refining existing reasoning steps. To overcome the limitations, we propose a new LLM reasoning framework, underlineReasoning viunderlinea underlinePlanning (RAP). RAP repurposes the LLM as both a world model and a reasoning agent, and incorporates a principled planning algorithm (based on Monto Carlo Tree Search) for strategic exploration in the vast reasoning space. During reasoning, the LLM (as agent) incrementally builds a reasoning tree under the guidance of the LLM (as world model) and task-specific rewards, and obtains a high-reward reasoning path efficiently with a proper balance between exploration textitvs. exploitation. We apply RAP to a variety of challenging reasoning problems including plan generation, math reasoning, and logical inference. Empirical results on these tasks demonstrate the superiority of RAP over various strong baselines, including CoT and least-to-most prompting with self-consistency. RAP on LLAMA-33B surpasses CoT on GPT-4 with 33% relative improvement in a plan generation setting.

Key facts. Cited by 846 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 46 of its 94 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

46 of 94 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models18 May
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul
  3. 2020Language Models are Few-Shot Learners28 May

Other references · 43

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
31 more
  1. 2018World Modelscited by 1,444
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  11. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  12. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  13. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  14. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  15. 2023Augmented Language Models: a Surveycited by 398
  16. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  17. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  18. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  19. 2023Faithful Chain-of-Thought Reasoningcited by 285
  20. 2022Generating Sequences by Learning to Self-Correctcited by 271
  21. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  22. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  23. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  24. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  25. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  26. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  27. 2017Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Drivingcited by 81
  28. 2018Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systemscited by 40
  29. 2022ALERT: Adapting Language Models to Reasoning Taskscited by 17
  30. 2023GPT is becoming a Turing machine: Here are some ways to program itcited by 16
  31. 2022Language Modeling with Latent Situationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 67; 2024: 244; 2025: 326; 2026: 209.

846 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  4. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  5. 2024A Survey on Large Language Models for Code Generationcited by 646
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  8. 2025LIMO: Less is More for Reasoningcited by 488
  9. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  10. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  11. 2023Cognitive Architectures for Language Agentscited by 422
  12. 2024Understanding the planning of LLM agents: A surveycited by 414
38 more
  1. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  2. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  6. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  7. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  9. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  10. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  11. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  12. 2024Token-Budget-Aware LLM Reasoningcited by 242
  13. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  14. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  15. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  16. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
  17. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  18. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  19. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  20. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  21. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  22. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  23. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  24. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  25. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  26. 2024Tree Search for Language Model Agentscited by 148
  27. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  28. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  29. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  30. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 141
  31. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  32. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  33. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  34. 2023Large Language Models for Robotics: A Surveycited by 132
  35. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  36. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  37. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 125
  38. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123

50 most cited of 846 All 846 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
  2. 2023Guiding Language Model Reasoning with Planning Tokens
  3. 2025LLMs Can Plan Only If We Tell Them
  4. 2025On the Limits of Innate Planning in Large Language Models
  5. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  6. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  7. 2024Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation
  8. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planning