mlsanityData to 7 Oct 2026

1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation

arXiv 2609.24432 ↗First posted 21 Sep 2026cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition. IER characterizes relative gradient estimation error under an optimal scalar baseline. A candidate-set approximation enables token selection based on IER and its combination with existing usefulness scores, while retaining the sampled reverse-KL training objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors in multiple settings, with sparse configurations matching or exceeding full OPD without token selection at small token budgets of 0.1%--1%. These results support accounting for both usefulness and gradient-estimation reliability when allocating sparse supervision. Our code is available at https://github.com/BruceSheng1202/IER-OPD.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

35 of 53 references.

References · 35

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  3. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  4. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  5. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  6. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 115
  7. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  8. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 57
  9. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 43
  10. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 42
  11. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 41
  12. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
23 more
  1. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  2. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 23
  3. 2026On the Position Bias of On-Policy Distillationcited by 23
  4. 2024Keypoint-based Progressive Chain-of-Thought Distillation for LLMscited by 22
  5. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 21
  6. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 20
  7. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  8. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  9. 2026Are Full Rollouts Necessary for On-Policy Distillation?cited by 14
  10. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  11. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 8
  12. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvementcited by 8
  13. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  14. 2026Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rolloutscited by 7
  15. 2026SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillationcited by 7
  16. 2026Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillationcited by 5
  17. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  18. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Modelscited by 4
  19. 2026Extremely Sparse Supervision Incentivizes Reasoning Abilitycited by 3
  20. 2026ClinAlign: Scaling Healthcare Alignment from Clinician Preferencecited by 2
  21. 2026Geometric Self-Distillation for Reasoning Generalizationcited by 2
  22. 2026Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoningcited by 2
  23. 2026Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Bettercited by 0