# When Can Model-Free Reinforcement Learning be Enough for Thinking? arXiv: https://arxiv.org/abs/2506.17124 ยท HTML: https://mlsanity.com/paper/2506.17124/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. - Builds on 33 of its 67 references that are in the map. - First posted on arXiv 20 Jun 2025. ## Placed in - LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/) - Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/) ## Builds on (most cited first, 30 of 33) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023) - [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024) - [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024) - [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024) - [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014) - [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023) - [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017) - [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023) - [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016) - [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017) - [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017) - [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024) - [An investigation of model-free planning](https://mlsanity.com/paper/1901.03559/) (2019) - [Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding](https://mlsanity.com/paper/2309.15028/) (2023) - [When is Tree Search Useful for LLM Planning? It Depends on the Discriminator](https://mlsanity.com/paper/2402.10890/) (2024) - [Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning](https://mlsanity.com/paper/2506.00318/) (2025) - [Learning to Search with MCTSnets](https://mlsanity.com/paper/1802.04697/) (2018) - [Learning to select computations](https://mlsanity.com/paper/1711.06892/) (2017) - [Interpreting Emergent Planning in Model-Free Reinforcement Learning](https://mlsanity.com/paper/2504.01871/) (2025) - [TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning](https://mlsanity.com/paper/1710.11417/) (2017) - [Reinforcement Learning Neural Turing Machines - Revised](https://mlsanity.com/paper/1505.00521/) (2015) - [Value Iteration Networks on Multiple Levels of Abstraction](https://mlsanity.com/paper/1905.11068/) (2019) ## Built on by (most cited first, 6 of 6) - [Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR](https://mlsanity.com/paper/2605.05965/) (2026) - [On the Role of Computation in Reinforcement Learning](https://mlsanity.com/paper/2602.05999/) (2026) - [When to Plan: Learning to Select Between Reactive Control and Deliberative Planning](https://mlsanity.com/paper/2607.16421/) (2026) - [Do Not Imitate, Reinforce: Iterative Classification via Belief Refinement](https://mlsanity.com/paper/2604.22110/) (2026) - [Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2606.25357/) (2026) - [What Pretraining and Midtraining Make Learnable from Rewards?](https://mlsanity.com/paper/2609.38446/) (2026) ## Abstract (arXiv) Recent work on large language models has demonstrated the use of model-free reinforcement learning (RL) to train reasoning-like capabilities. The emergence of "thinking" through model-free RL is interesting as thinking actions neither produce reward nor change the external world state to one where the agent is more likely to get reward. This paper seeks to build a domain-independent understanding of when model-free RL will lead to such "thinking" as a strategy for reward maximization. To build this understanding, we first introduce a theoretical model which we call a thought Markov decision process (MDP). Thought MDPs minimally extend the classical MDP model to include an abstract notion of thought state and thought action. Using the thought MDP model, we prove the importance of policy initialization in determining whether or not thinking emerges and show formally that thought actions are equivalent to the agent choosing to perform a step of policy improvement before continuing to act. We then show that open-source LLMs satisfy the conditions that our theory predicts are necessary for model-free RL to produce thinking-like behavior. Finally, we hypothesize sufficient conditions that would enable thinking to be learned outside of language generation and introduce a toy domain where a combination of multi-task pre-training and designated thought actions enable more data-efficient RL compared to non-thinking agents.