mlsanityData to 7 Oct 2026

GATES: Self-Distillation under Privileged Context with Consensus Gating

arXiv 2602.20574 ↗First posted 24 Feb 2026cs.LG cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0% to 62.0%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2% to 35.4%.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

20 of 31 references.

References · 20

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  10. 2024Self-Rewarding Language Modelscited by 654
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
8 more
  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  3. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  4. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  5. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  6. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  7. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  8. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 30.

30 papers.

Sort:

By area:

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  2. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  3. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  4. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 20
  5. 2026Trajectory-Refined Distillationcited by 14
  6. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  7. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoningcited by 9
  8. 2026DOPD: Dual On-policy Distillationcited by 9
  9. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  10. 2026OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Modelscited by 6
  11. 2026Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agentscited by 6
  12. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillationcited by 6
18 more
  1. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 6
  2. 2026Rethinking Privileged Information in On-Policy Self-Distillationcited by 6
  3. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  4. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillationcited by 5
  5. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillationcited by 4
  6. 2026Consensus as Privileged Context for Label-Free Self-Distillationcited by 3
  7. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillationcited by 3
  8. 2026Geometric Self-Distillation for Reasoning Generalizationcited by 2
  9. 2026The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillationcited by 2
  10. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  11. 2026Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifactscited by 1
  12. 2026Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repaircited by 0
  13. 2026Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrastcited by 0
  14. 2026SR-OPSD: Self-Referenced On-Policy Self-Distillationcited by 0
  15. 2026Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoningcited by 0
  16. 2026OSPD: On-Policy Self-Distillation for Persona-Consistent Dialoguecited by 0
  17. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Modelscited by 0
  18. 2026Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Skill-Conditioned Gated Self-Distillation for LLM Reasoning
  2. 2026Activation-Conditioned Self-Distillation
  3. 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
  4. 2026GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning
  5. 2026Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning
  6. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
  7. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think
  8. 2026E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation