mlsanityData to 7 Oct 2026

Recursive Self-Improvement via On-Policy Distillation for Reasoning

arXiv 2609.30652 ↗First posted 25 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher's next-token predictions. This provides dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE). Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model's own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL). Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

31 of 39 references.

References · 31

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
19 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  3. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  4. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  7. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  8. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  9. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  10. 2026Self-Distilled RLVRcited by 109
  11. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
  12. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 65
  13. 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
  14. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  15. 2025The Geometry of Self-Verification in a Task-Specific Reasoning Modelcited by 12
  16. 2025Aligning Large Language Models via Fully Self-Synthetic Datacited by 3
  17. 2025Scaling Reinforcement Learning for Content Moderation with Large Language Modelscited by 3
  18. 2025PIKA: Expert-Level Synthetic Datasets for Post-Training Alignment from Scratchcited by 1
  19. 2026GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zerocited by 1

Built on by

Not cited yet.