# Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information

arXiv: https://arxiv.org/abs/2605.11609 · HTML: https://mlsanity.com/paper/2605.11609/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 22 of 22)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://mlsanity.com/paper/2605.00674/) (2026)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Olmo 3](https://mlsanity.com/paper/2512.13961/) (2025)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [How Far Can Unsupervised RLVR Scale LLM Training?](https://mlsanity.com/paper/2603.08660/) (2026)
- [PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence](https://mlsanity.com/paper/2603.11178/) (2026)
- [Efficient Process Reward Modeling via Contrastive Mutual Information](https://mlsanity.com/paper/2604.10660/) (2026)

## Built on by (most cited first, 14 of 14)

- [RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation](https://mlsanity.com/paper/2606.11709/) (2026)
- [A Formula-Driven Survey and Research Agenda for On-Policy Distillation](https://mlsanity.com/paper/2606.22793/) (2026)
- [Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation](https://mlsanity.com/paper/2607.10805/) (2026)
- [β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation](https://mlsanity.com/paper/2607.28582/) (2026)
- [Negative Self-Distillation: Learning to Reason by Avoiding Flaws](https://mlsanity.com/paper/2609.11699/) (2026)
- [Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts](https://mlsanity.com/paper/2606.10334/) (2026)
- [Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast](https://mlsanity.com/paper/2608.08764/) (2026)
- [Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents](https://mlsanity.com/paper/2608.09555/) (2026)
- [On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation](https://mlsanity.com/paper/2609.21561/) (2026)
- [One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation](https://mlsanity.com/paper/2608.25936/) (2026)
- [VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.28306/) (2026)
- [iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs](https://mlsanity.com/paper/2609.24646/) (2026)
- [Privileged Context as Drift in On-Policy Self-Distillation](https://mlsanity.com/paper/2610.07842/) (2026)
- [Self-Distillation with Verbal Regularization](https://mlsanity.com/paper/2610.16354/) (2026)

## Abstract (arXiv)

On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.