mlsanityData to 14 Oct 2026

Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration

arXiv 2508.16677 ↗First posted 21 Aug 2025cs.LG cs.AI cs.CLcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Many existing studies have achieved significant improvements in the reasoning capabilities of large language models (LLMs) through reinforcement learning with verifiable rewards (RLVR), while the enhancement of reasoning abilities in small language models (SLMs) has not yet been sufficiently explored. Combining distilled data from larger models with RLVR on small models themselves is a natural approach, but it still faces various challenges and issues. Therefore, we propose textitunderlineRecall-textitunderlineExtend textitunderlineDynamics(RED): Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration. In this paper, we explore the perspective of varying exploration spaces, balancing offline distillation with online reinforcement learning. Simultaneously, we specifically design and optimize for the insertion problem within offline data. By monitoring the ratio of entropy changes in the model concerning offline and online data, we regulate the weight of offline-SFT, thereby addressing the issues of insufficient exploration space in small models and the redundancy and complexity during the distillation process. Furthermore, to tackle the distribution discrepancies between offline data and the current policy, we design a sample-accuracy-based policy shift mechanism that dynamically chooses between imitating offline distilled data and learning from its own policy.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 32 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

28 of 32 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  2. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  3. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  4. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  5. 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning30 May
  6. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning23 May
  7. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning20 May
  8. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs19 May
  9. 2025Learning to Reason under Off-Policy Guidance21 Apr
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
5 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2024Process Reward Model with Q-Value Rankings15 Oct
  4. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  6. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  7. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  8. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  9. 2025Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcementcited by 14
  10. 2024UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Functioncited by 9
  11. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoningcited by 9
  3. 2026From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  2. 2025Reasoning with Exploration: An Entropy Perspective
  3. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
  4. 2026ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning
  5. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  6. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
  7. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  8. 2025RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models