mlsanityData to 7 Oct 2026

Thinking Fast and Slow with Deep Learning and Tree Search

arXiv 1705.08439 ↗First posted 23 May 2017cs.AI cs.LGcited by 245

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Sequential decision making problems, such as structured prediction, robotic control, and game playing, require a combination of planning policies and generalisation of those plans. In this paper, we present Expert Iteration (ExIt), a novel reinforcement learning algorithm which decomposes the problem into separate planning and generalisation tasks. Planning new policies is performed by tree search, while a deep neural network generalises those plans. Subsequently, tree search is improved by using the neural network policy to guide search, increasing the strength of new plans. In contrast, standard deep Reinforcement Learning algorithms rely on a neural network not only to generalise plans, but to discover them too. We show that ExIt outperforms REINFORCE for training a neural network to play the board game Hex, and our final tree search agent, trained tabula rasa, defeats MoHex 1.0, the most recent Olympiad Champion player to be publicly released.

Key facts. Cited by 245 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 0 of its 23 references that are in the map. First posted on arXiv 23 May 2017.

Builds on

0 of 23 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 1; 2021: 27; 2022: 33; 2023: 32; 2024: 52; 2025: 53; 2026: 45.

245 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  5. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  9. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  10. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
38 more
  1. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  2. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  3. 2026Reinforcement Learning via Self-Distillationcited by 285
  4. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  5. 2022Autoformalization with Large Language Modelscited by 252
  6. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  7. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  8. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  9. 2022TALM: Tool Augmented Language Modelscited by 197
  10. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  11. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  12. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  13. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 158
  14. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  15. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  16. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  17. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  18. 2025Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Provingcited by 126
  19. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  20. 2024Lean Workbook: A large-scale Lean problem set formalized from natural language math problemscited by 115
  21. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  22. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  23. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  24. 2025BFS-Prover: Scalable Best-First Tree Search for LLM-based Automatic Theorem Provingcited by 82
  25. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 78
  26. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 77
  27. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  28. 2025Self-rewarding correction for mathematical reasoningcited by 67
  29. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  30. 2022A Survey on Model-based Reinforcement Learningcited by 65
  31. 2024Lean-STaR: Learning to Interleave Thinking and Provingcited by 64
  32. 2021Scaling Scaling Laws with Board Gamescited by 62
  33. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  34. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
  35. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  36. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 58
  37. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  38. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48

50 most cited of 245 All 245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing Interpretable Decision Tree Policies for Reinforcement Learning
  2. 2020Tracking the Race Between Deep Reinforcement Learning and Imitation Learning -- Extended Version
  3. 2024Mastering Board Games by External and Internal Planning with Language Models
  4. 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoning
  5. 2025Learning Game-Playing Agents with Generative Code Optimization
  6. 2023Forecaster: Towards Temporally Abstract Tree-Search Planning from Pixels
  7. 2018Verifiable Reinforcement Learning via Policy Extraction
  8. 2025Policy Guided Tree Search for Enhanced LLM Reasoning