# AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence

arXiv: https://arxiv.org/abs/2502.13943 · HTML: https://mlsanity.com/paper/2502.13943/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 41 of its 48 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 30 of 41)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training](https://mlsanity.com/paper/2309.17179/) (2023)
- [Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning](https://mlsanity.com/paper/2410.08146/) (2024)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://mlsanity.com/paper/2402.08983/) (2024)
- [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023)
- [Free Process Rewards without Process Labels](https://mlsanity.com/paper/2412.01981/) (2024)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [Let's reward step by step: Step-Level reward model as the Navigators for Reasoning](https://mlsanity.com/paper/2310.10080/) (2023)
- [LLMs cannot find reasoning errors, but can correct them given the error location](https://mlsanity.com/paper/2311.08516/) (2023)

## Built on by (most cited first, 24 of 24)

- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning](https://mlsanity.com/paper/2505.11896/) (2025)
- [LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs](https://mlsanity.com/paper/2504.14655/) (2025)
- [A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models](https://mlsanity.com/paper/2510.08049/) (2025)
- [A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law](https://mlsanity.com/paper/2505.02665/) (2025)
- [Guiding Reasoning in Small Language Models with LLM Assistance](https://mlsanity.com/paper/2504.09923/) (2025)
- [AdapThink: Adaptive Thinking Preferences for Reasoning Language Model](https://mlsanity.com/paper/2506.18237/) (2025)
- [RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models](https://mlsanity.com/paper/2602.17053/) (2026)
- [Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey](https://mlsanity.com/paper/2510.01925/) (2025)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment](https://mlsanity.com/paper/2605.03327/) (2026)
- [Recurrent Confidence Chain: Temporal-Aware Uncertainty Quantification in Large Language Models](https://mlsanity.com/paper/2601.13368/) (2026)
- [PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning](https://mlsanity.com/paper/2605.13467/) (2026)
- [Mitigating Overthinking through Reasoning Shaping](https://mlsanity.com/paper/2510.09535/) (2025)
- [From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models](https://mlsanity.com/paper/2601.15690/) (2026)
- [Reward-Driven Interaction: Enhancing Proactive Dialogue Agents through User Satisfaction Prediction](https://mlsanity.com/paper/2505.18731/) (2025)
- [Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimization](https://mlsanity.com/paper/2510.11104/) (2025)
- [Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learning](https://mlsanity.com/paper/2603.18533/) (2026)
- [TRACES: Tagging Reasoning Steps for Adaptive Cost-Efficient Early-Stopping](https://mlsanity.com/paper/2604.21057/) (2026)
- [Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning](https://mlsanity.com/paper/2609.11061/) (2026)
- [Unlocking Token Rewards via Training-Free Reward Attribution](https://mlsanity.com/paper/2610.18704/) (2026)

## Abstract (arXiv)

Current approaches for training Process Reward Models (PRMs) often involve breaking down responses into multiple reasoning steps using rule-based techniques, such as using predefined placeholder tokens or setting the reasoning step's length into a fixed size. These approaches overlook the fact that specific words do not typically mark true decision points in a text. To address this, we propose AdaptiveStep, a method that divides reasoning steps based on the model's confidence in predicting the next word. This division method provides more decision-making information at each step, enhancing downstream tasks, such as reward model learning. Moreover, our method does not require manual annotation. We demonstrate its effectiveness through experiments with AdaptiveStep-trained PRMs in mathematical reasoning and code generation tasks. Experimental results indicate that the outcome PRM achieves state-of-the-art Best-of-N performance, surpassing greedy search strategy with token-level value-guided decoding, while also reducing construction costs by over 30% compared to existing open-source PRMs. In addition, we provide a thorough analysis and case study on the PRM's performance, transferability, and generalization capabilities.