mlsanityData to 14 Oct 2026

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

arXiv 2609.33200 ↗First posted 27 Sep 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning also fits: Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

On-policy self-distillation trains reasoning models on their own trajectories using dense token distribution guidance from a privileged teacher with access to a verified solution. This supervision transfers what the teacher predicts without directly transferring where it attends within the preceding context. We introduce On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation. Because the privileged teacher can attend to verified solution tokens unavailable to the student, OPASD projects teacher attention onto student-visible positions and renormalizes the resulting distribution before alignment. Across three model sizes and four competition-level mathematics benchmarks, OPASD consistently outperforms token-only OPSD, improving average accuracy by 4.98 to 8.40 percentage points. OPASD also avoids the response-length inflation and performance degradation observed with token-only distillation, reducing generated rollout tokens by 73.9% and estimated model compute by 72.6% while training 1.53x faster. These results show that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 57 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

52 of 57 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026OISD: On-Policy Internal Self-Distillation of Language Models27 May
  2. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning12 May
  3. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information17 Apr
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  5. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  6. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation11 Mar
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 45

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
33 more
  1. 2016Sequence-Level Knowledge Distillationcited by 801
  2. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  3. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  4. 2026Reinforcement Learning via Self-Distillationcited by 332
  5. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  6. 2026Self-Distillation Enables Continual Learningcited by 247
  7. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  8. 2026On-Policy Context Distillation for Language Modelscited by 186
  9. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  10. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  11. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  12. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  13. 2022Learning by Distilling Contextcited by 138
  14. 2026Self-Distilled RLVRcited by 131
  15. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  16. 2026Privileged Information Distillation for Language Modelscited by 90
  17. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  18. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  19. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 53
  20. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  21. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 45
  22. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  23. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 35
  24. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 35
  25. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 34
  26. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 18
  27. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 14
  28. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  29. 2026OPRD: On-Policy Representation Distillationcited by 11
  30. 2023How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectivescited by 9
  31. 2026Reinforced Attention Learningcited by 6
  32. 2024Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translationcited by 5
  33. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning
  2. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
  3. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  4. 2026E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
  5. 2026Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation
  6. 2026Activation-Conditioned Self-Distillation
  7. 2026Rethinking Privileged Information in On-Policy Self-Distillation
  8. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models