# Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

arXiv: https://arxiv.org/abs/2605.10781 · HTML: https://mlsanity.com/paper/2605.10781/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 30)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models](https://mlsanity.com/paper/2508.10751/) (2025)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining](https://mlsanity.com/paper/2504.07912/) (2025)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Outcome-based Exploration for LLM Reasoning](https://mlsanity.com/paper/2509.06941/) (2025)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models](https://mlsanity.com/paper/2512.07783/) (2025)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective](https://mlsanity.com/paper/2510.10150/) (2025)
- [Diversity-Incentivized Exploration for Versatile Reasoning](https://mlsanity.com/paper/2509.26209/) (2025)
- [Revisiting Entropy in Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2511.05993/) (2025)
- [Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs](https://mlsanity.com/paper/2603.22446/) (2026)
- [Improving RL Exploration for LLM Reasoning through Retrospective Replay](https://mlsanity.com/paper/2504.14363/) (2025)
- [Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization](https://mlsanity.com/paper/2602.23008/) (2026)
- [The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models](https://mlsanity.com/paper/2510.02230/) (2025)
- [The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View](https://mlsanity.com/paper/2510.04028/) (2025)
- [Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models](https://mlsanity.com/paper/2509.26114/) (2025)
- [SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training](https://mlsanity.com/paper/2510.08141/) (2025)
- [DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning](https://mlsanity.com/paper/2602.19895/) (2026)

## Built on by (most cited first, 15 of 15)

- [RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation](https://mlsanity.com/paper/2606.11709/) (2026)
- [UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation](https://mlsanity.com/paper/2606.29502/) (2026)
- [Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR](https://mlsanity.com/paper/2609.04108/) (2026)
- [Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation](https://mlsanity.com/paper/2607.10805/) (2026)
- [CriPO: Enhancing Rubric-based RL via Self-Distillation](https://mlsanity.com/paper/2607.18082/) (2026)
- [LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks](https://mlsanity.com/paper/2607.18110/) (2026)
- [Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs](https://mlsanity.com/paper/2609.02548/) (2026)
- [CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO](https://mlsanity.com/paper/2606.00172/) (2026)
- [Negative Self-Distillation: Learning to Reason by Avoiding Flaws](https://mlsanity.com/paper/2609.11699/) (2026)
- [Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback](https://mlsanity.com/paper/2606.14368/) (2026)
- [On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation](https://mlsanity.com/paper/2609.21561/) (2026)
- [FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience](https://mlsanity.com/paper/2609.03241/) (2026)
- [Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning](https://mlsanity.com/paper/2609.33781/) (2026)
- [A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.13744/) (2026)
- [RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning](https://mlsanity.com/paper/2610.17623/) (2026)

## Abstract (arXiv)

Self-distillation has emerged as a powerful framework for post-training LLMs, where a teacher conditioned on extra information guides a student without it, both from the same model. While this guidance is useful when the student has failed, on successful rollouts, the same mechanism instead overwrites the student's choices and suppresses it's own reasoning. Therefore, we propose reading the original self-distillation signal in reverse: when the student succeeds along a path the teacher would not have predicted, these tokens reflect its self-driven reasoning. Building on this, we propose RLRT (RLVR with Reversed Teacher), which augments GRPO by reinforcing these tokens on correct rollouts. We interpret this as a new form of exploration in RLVR: not uniform diversity, but valuable exploration grounded in the student's own success. Across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints, RLRT substantially outperforms self-distillation and exploration-based baselines, establishing information asymmetry as a new, principled design axis for RLVR.