mlsanityData to 7 Oct 2026

ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery

arXiv 2610.11332 ↗First posted 8 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

31 of 47 references.

References · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2017Decoupled Weight Decay Regularizationcited by 1,722
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  9. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  10. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  11. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  12. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
19 more
  1. 2025Generalizing Verifiable Instruction Followingcited by 178
  2. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  3. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  4. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  5. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  6. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  7. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  8. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  9. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  10. 2023The LLM Surgeoncited by 36
  11. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 35
  12. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  13. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  14. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 7
  15. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  16. 2025Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Modelscited by 4
  17. 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scalingcited by 2
  18. 2026Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initializationcited by 2
  19. 2026MuCRASP: Multimodal Chain-of-thought Reasoning aware Structured Pruningcited by 1

Built on by

Not cited yet.