Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Injecting new reasoning knowledge into Large Language Models (LLMs) via post-training often induces catastrophic forgetting. Recent studies emphasize the importance of on-policy data but suggest that KL-divergence fails to mitigate forgetting. In contrast, we show, both analytically and empirically, that the KL-constrained reward formulation actually plays a critical role in retaining knowledge during post-training. This motivates our Surgical Post-Training (SPOT), a proximal on-policy distillation framework designed to optimize reasoning efficiently while preserving prior knowledge. SPOT consists of (1) a data rectification pipeline employing an Oracle to surgically correct erroneous steps via minimal edits, generating proximal on-policy data; and (2) a reward-based binary cross-entropy objective essential for enhancing reasoning and mitigating forgetting. Empirically, with only 4k rectified math pairs, SPOT improves Qwen3-8B's accuracy by 6.2% on average across in-domain and out-of-domain tasks, requiring merely 16-minute model training on 8x H800 GPUs. Moreover, SPOT provides a superior initialization for subsequent reinforcement learning, significantly elevating the performance ceiling. Code: https://github.com/Visual-AI/SPoT
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 39 of its 45 references that are in the map. First posted on arXiv 2 Mar 2026.
Builds on
39 of 45 references.
References · 39
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
27 more
- 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
- 2016Sequence-Level Knowledge Distillationcited by 761
- 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
- 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
- 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
- 2015Policy Distillationcited by 414
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
- 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
- 2024Iterative Reasoning Preference Optimizationcited by 244
- 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
- 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
- 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
- 2026MiMo-V2-Flash Technical Reportcited by 163
- 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
- 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
- 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
- 2024Learning Dynamics of LLM Finetuningcited by 93
- 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
- 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
- 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
- 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
- 2024Binary Classifier Optimization for Large Language Model Alignmentcited by 44
- 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learningcited by 35
- 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Gamescited by 16
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation
- 2026Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning
- 2026Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning
- 2025Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models
- 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models
- 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
- 2026Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA