# Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement

arXiv: https://arxiv.org/abs/2505.07961 · HTML: https://mlsanity.com/paper/2505.07961/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 12 May 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 20 of 20)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2504.01296/) (2025)
- [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024)
- [Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning](https://mlsanity.com/paper/2502.18080/) (2025)
- [Small Models Struggle to Learn from Strong Reasoners](https://mlsanity.com/paper/2502.12143/) (2025)
- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [Scaling Test-Time Compute Without Verification or RL is Suboptimal](https://mlsanity.com/paper/2502.12118/) (2025)
- [Following Length Constraints in Instructions](https://mlsanity.com/paper/2406.17744/) (2024)
- [Scalable Chain of Thoughts via Elastic Reasoning](https://mlsanity.com/paper/2505.05315/) (2025)
- [Test-Time Training Provably Improves Transformers as In-context Learners](https://mlsanity.com/paper/2503.11842/) (2025)
- [Precise Length Control in Large Language Models](https://mlsanity.com/paper/2412.11937/) (2024)

## Built on by (most cited first, 14 of 14)

- [BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning](https://mlsanity.com/paper/2506.17211/) (2025)
- [Continuous Chain of Thought Enables Parallel Exploration and Reasoning](https://mlsanity.com/paper/2505.23648/) (2025)
- [Training Large Language Models To Reason In Parallel With Global Forking Tokens](https://mlsanity.com/paper/2510.05132/) (2025)
- [Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration](https://mlsanity.com/paper/2508.16677/) (2025)
- [Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning](https://mlsanity.com/paper/2510.21398/) (2025)
- [LiveThinking: Enabling Real-Time Efficient Reasoning for AI-Powered Livestreaming via Reinforcement Learning](https://mlsanity.com/paper/2510.07685/) (2025)
- [A State-Transition Framework for Efficient LLM Reasoning](https://mlsanity.com/paper/2602.01198/) (2026)
- [SmartChunk Retrieval: Query-Aware Chunk Compression with Planning for Efficient Document RAG](https://mlsanity.com/paper/2602.22225/) (2025)
- [Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models](https://mlsanity.com/paper/2607.12771/) (2026)
- [From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation](https://mlsanity.com/paper/2607.19395/) (2026)
- [VSPO: Vector-Steered Policy Optimization for Behavioral Control](https://mlsanity.com/paper/2605.15604/) (2026)
- [OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping](https://mlsanity.com/paper/2607.11089/) (2026)
- [Rewarding Novel Deductions: Solver-guided Process Supervision for Logical Reasoning](https://mlsanity.com/paper/2609.34660/) (2026)
- [Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR](https://mlsanity.com/paper/2610.16105/) (2026)

## Abstract (arXiv)

Recent research enhances language model reasoning by scaling test-time compute via longer chain-of-thought traces. This often improves accuracy but also introduces redundancy and high computational cost, especially for small language models distilled with supervised fine-tuning (SFT). In this work, we propose new algorithms to improve token-efficient reasoning with small-scale models by effectively trading off accuracy and computation. We first show that the post-SFT model fails to determine the optimal stopping point of the reasoning process, resulting in verbose and repetitive outputs. Verbosity also significantly varies across wrong vs correct responses. To address these issues, we propose two solutions: (1) Temperature scaling (TS) to control the stopping point for the thinking phase and thereby trace length, and (2) TLDR: a length-regularized reinforcement learning method based on GRPO that facilitates multi-level trace length control (e.g. short, medium, long reasoning). Experiments on four reasoning benchmarks, MATH500, AMC, AIME24 and OlympiadBench, demonstrate that TS is highly effective compared to s1's budget forcing approach and TLDR significantly improves token efficiency by about 50% with minimal to no accuracy loss over the SFT baseline. Moreover, TLDR also facilitates flexible control over the response length, offering a practical and effective solution for token-efficient reasoning in small models. Ultimately, our work reveals the importance of stopping time control, highlights shortcomings of pure SFT, and provides effective algorithmic recipes.