mlsanityData to 14 Oct 2026

Beyond Imitation: Auditing the Recoverability of Reasoning in Distilled Models

arXiv 2609.26216 ↗First posted 11 Aug 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

A correct teacher solution becomes useful supervision when the receiving student can continue its reasoning. We measure this compatibility with prefix recovery: after revealing 25%, 50%, or 75% of a verified solution, we test whether the student completes it correctly. We connect recovery to the cosine conflict between cross-entropy and reverse-KL gradients over the full vocabulary. Across adjacent Qwen3 teacher-student pairs from 0.6B to 8B parameters, reverse-KL distillation delivers its most consistent mathematical and code improvements for the two students below 2B parameters. On a fixed cohort of 1,000 trajectories, average prefix recovery rises from 71.0% to 91.9% as student size increases from 0.6B to 4B, and the robust-fragile recovery gap contracts from 46.0 to 14.4 percentage points. With the teacher fixed at 8B, conflict separation falls from 0.993 to 0.233. An independent objective intervention finds the largest reverse-KL rescue on fragile trajectories. The three measurements locate the same capacity-dependent transfer regime: distribution matching has the greatest headroom when correct traces remain unevenly recoverable. Prefix recovery provides a practical diagnostic for selecting costly distribution-level distillation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 31 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

30 of 31 references.

References · 30

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
18 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2016Sequence-Level Knowledge Distillationcited by 801
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  8. 2018Born Again Neural Networkscited by 546
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2023A Survey on Model Compression for Large Language Modelscited by 269
  12. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  13. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  14. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  15. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  16. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  17. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  18. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026What Will Post-Training Fix? Per-Problem Gains Are Shared Across Independent RL Runs, and Existing Checkpoints Predict Them Better Than A Priori Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
  2. 2026Rethinking Reverse KL as Adaptive Entropy Distillation
  3. 2026Outcome-Confounded Local Supervision in On-Policy Distillation
  4. 2025UNDO: Understanding Distillation as Optimization
  5. 2026Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation
  6. 2026On-Policy Distillation Teaches New Skills but Not New Knowledge
  7. 2026Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
  8. 2026Reward-Aligned Reweighting for On-Policy Distillation