mlsanityData to 7 Oct 2026

Thinker: Learning to Plan and Act

arXiv 2307.14993 ↗First posted 27 Jul 2023cs.AI cs.LGcited by 13

On the map: World models → World models for language agents

Abstract (arXiv)

We propose the Thinker algorithm, a novel approach that enables reinforcement learning agents to autonomously interact with and utilize a learned world model. The Thinker algorithm wraps the environment with a world model and introduces new actions designed for interacting with the world model. These model-interaction actions enable agents to perform planning by proposing alternative plans to the world model before selecting a final action to execute in the environment. This approach eliminates the need for handcrafted planning algorithms by enabling the agent to learn how to plan autonomously and allows for easy interpretation of the agent's plan with visualization. We demonstrate the algorithm's effectiveness through experimental results in the game of Sokoban and the Atari 2600 benchmark, where the Thinker algorithm achieves state-of-the-art performance and competitive results, respectively. Visualizations of agents trained with the Thinker algorithm demonstrate that they have learned to plan effectively with the world model to select better actions. Thinker is the first work showing that an RL agent can learn to plan with a learned world model in complex environments.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 34 of its 57 references that are in the map. First posted on arXiv 27 Jul 2023.

Builds on

34 of 57 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Layer Normalizationcited by 5,579
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2023Mastering Diverse Domains through World Modelscited by 1,193
  8. 2020Mastering Atari with Discrete World Modelscited by 990
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
22 more
  1. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  2. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  3. 2016Learning to reinforcement learncited by 369
  4. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  5. 2018Distributed Prioritized Experience Replaycited by 260
  6. 2021Mastering Atari Games with Limited Datacited by 235
  7. 2016Value Iteration Networkscited by 158
  8. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  9. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  10. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  11. 2017Value Prediction Networkcited by 117
  12. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  13. 2022A Survey on Model-based Reinforcement Learningcited by 65
  14. 2019An investigation of model-free planningcited by 64
  15. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  16. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  17. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  18. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  19. 2018Learning to Search with MCTSnetscited by 32
  20. 2017Learning model-based planning from scratchcited by 21
  21. 2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
  22. 2019Policy Gradient Search: Online Planning and Expert Iteration without Search Treescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  2. 2024Planning in a recurrent neural network that plays Sokobancited by 17
  3. 2025Learning from Failures in Multi-Attempt Reinforcement Learningcited by 6
  4. 2024Predicting Future Actions of Reinforcement Learning Agentscited by 4
  5. 2025Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environmentscited by 4
  6. 2026Finding the Time to Think: Learning Planning Budgets in Real-Time RLcited by 2
  7. 2026Reinforced Planning with Latent World Modelscited by 2
  8. 2026Temporal Recurrence Favors Fewer Layerscited by 1
  9. 2025Thinking agents for zero-shot generalization to qualitatively novel taskscited by 0
  10. 2025Tackling GNARLy Problems: Graph Neural Algorithmic Reasoning Reimagined through Reinforcement Learningcited by 0
  11. 2026Reinforcement Learning with Multi-Step Lookahead Information Via Adaptive Batchingcited by 0
  12. 2026Beyond Static Graph World Models: Learning Stochastic Latent Dynamics over Evolving Topologiescited by 0
1 more
  1. 2026Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models
  2. 2026UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning
  3. 2023Forecaster: Towards Temporally Abstract Tree-Search Planning from Pixels
  4. 2026APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents
  5. 2025General Agentic Planning Through Simulative Reasoning with World Models
  6. 2025RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policy
  7. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
  8. 2025Synthesizing world models for bilevel planning