mlsanityData to 14 Oct 2026

The Role of Feedback Alignment in Self-Distillation

arXiv 2606.11173 ↗First posted 9 Jun 2026cs.AI cs.LGcited by 4

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

Conditioning a language model on additional context, such as feedback on a previous attempt, typically improves its response. Self-distillation trains the model to retain this improvement when the context is not present. The method works by matching the model's output distribution under two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns therefore depends on what context the self-teacher receives, yet the design of this context remains largely unexplored. We study context design for self-distillation by training a solver on feedback from a frozen critic. We compare three conditions: (i) a binary reward (GRPO), (ii) the reference solution, and (iii) a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and reference-solution-conditioned self-distillation by 5.27 points (Avg@12). Per-token advantage analysis reveals why: step-aligned feedback targets only the tokens where reasoning fails, leaving correct behavior intact. Conditioning on the reference solution, by contrast, pressures the model to change its behavior at every token (even correct steps) because an alternative derivation inevitably differs in phrasing and approach. This suggests that structural alignment between feedback and the solver's reasoning is a key driver of self-distillation effectiveness.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 29 of its 33 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

29 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Reinforcement Learning via Self-Distillation28 Jan
  2. 2026Self-Distillation Enables Continual Learning27 Jan
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2025Qwen3 Technical Reportcited by 8,820
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
14 more
  1. 2022In-context Learning and Induction Headscited by 964
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  5. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  8. 2024Self-Rewarding Language Modelscited by 681
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  10. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  11. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  12. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  13. 2026Aligning Language Models from User Interactionscited by 28
  14. 2022Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 35
  2. 2026What Does Privileged Information Add to On-Policy Self-Distillation?cited by 2
  3. 2026One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillationcited by 0
  4. 2026UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention
  2. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
  3. 2026Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models
  4. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
  5. 2026From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation
  6. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  7. 2026A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
  8. 2026Rethinking Privileged Information in On-Policy Self-Distillation