mlsanityData to 14 Oct 2026

OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning

arXiv 2605.12400 ↗First posted 12 May 2026cs.LG cs.AIcited by 16

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories. Despite its promise, OPSD can suffer from training instability due to a pattern mismatch between teacher and student responses. Self-reflected teacher responses may introduce reflection-induced biases and response templates that miscalibrate token-level supervision, ultimately harming the student's reasoning ability. To mitigate this issue, we propose OGLS-SD, an outcome-guided logit-steering framework that leverages verifiable outcome rewards to calibrate privileged teacher logits. Specifically, OGLS-SD contrasts teacher logits induced by successful and failed on-policy trajectories, constructing an outcome-discriminative steering direction for token-level guidance. Experiments on mathematical reasoning benchmarks show that OGLS-SD stabilizes self-distillation and improves performance over standard OPSD and other variants.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 46 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

22 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  2. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  6. 2025OpenAI GPT-5 System Cardcited by 989
  7. 2023Steering Language Models With Activation Engineeringcited by 922
  8. 2025Group Sequence Policy Optimizationcited by 835
  9. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  10. 2026Reinforcement Learning via Self-Distillationcited by 332
  11. 2026MiMo-V2-Flash Technical Reportcited by 191
  12. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
7 more
  1. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  2. 2026Self-Distilled RLVRcited by 131
  3. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  4. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  5. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  6. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  7. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Modelscited by 5
  2. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  3. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillationcited by 3
  4. 2026Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spotscited by 2
  5. 2026Adaptive Supervised Anchoring for On-Policy Self-Distillationcited by 1
  6. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agentscited by 1
  7. 2026Activation-Conditioned Self-Distillationcited by 1
  8. 2026When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillationcited by 1
  9. 2026Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoningcited by 0
  10. 2026SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillationcited by 0
  11. 2026Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoningcited by 0
  12. 2026Inductive Feedback for Mixed-Policy Distillationcited by 0
4 more
  1. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Modelscited by 0
  2. 2026REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reusecited by 0
  3. 2026PreSO: Privileged Representation Isolation for On-Policy Self-Distillationcited by 0
  4. 2026Recovering Reliable Supervision via Critique-Guided On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
  2. 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
  3. 2026Skill-Conditioned Gated Self-Distillation for LLM Reasoning
  4. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
  5. 2026Geometric Self-Distillation for Reasoning Generalization
  6. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think
  7. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning
  8. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning