# Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold

arXiv: https://arxiv.org/abs/2607.27770 · HTML: https://mlsanity.com/paper/2607.27770/

## Key facts

- Cited by 1 papers in the map (data through 7 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Builds on (most cited first, 24 of 24)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Skywork Open Reasoner 1 Technical Report](https://mlsanity.com/paper/2505.22312/) (2025)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [TIP: Token Importance in On-Policy Distillation](https://mlsanity.com/paper/2604.14084/) (2026)
- [MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training](https://mlsanity.com/paper/2606.30406/) (2026)
- [Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2402.04616/) (2024)
- [OPSDL: On-Policy Self-Distillation for Long-Context Language Models](https://mlsanity.com/paper/2604.17535/) (2026)
- [ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains](https://mlsanity.com/paper/2605.28014/) (2026)
- [Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review](https://mlsanity.com/paper/2410.03663/) (2024)
- [LLM-Oriented Token-Adaptive Knowledge Distillation](https://mlsanity.com/paper/2510.11615/) (2025)
- [Self-Supervised On-Policy Distillation for Reasoning Language Models](https://mlsanity.com/paper/2605.17497/) (2026)
- [Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs](https://mlsanity.com/paper/2602.01064/) (2026)
- [EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer](https://mlsanity.com/paper/2605.01732/) (2026)
- [SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision](https://mlsanity.com/paper/2606.28562/) (2026)

## Built on by (most cited first, 1 of 1)

- [Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning](https://mlsanity.com/paper/2609.15404/) (2026)

## Abstract (arXiv)

A single reinforcement-learning run can produce a strong reasoner yet an incomplete teacher: it often amplifies only a subset of the valid solution modes. We argue that reinforcement learning (RL)-trained policies should therefore be viewed as local probes of a multi-basin reasoning solution manifold, rather than as globally reliable supervisors. Based on this view, we propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation. In the expansion stage, Residual Group Relative Policy Optimization (RGRPO) trains a sequence of teachers from a common initialization and redirects each later round toward examples not yet covered by the accumulated teacher union. In the compression stage, reliability-gated Teacher-Union On-policy Distillation (TU-OPD) lets the student learn from its own response prefixes. For each example, only reliable teachers contribute, and their sampled-token OPD losses are weighted by their per-example quality. We further introduce Consensus-Residual Decomposition, which preserves a winner teacher's excess token preferences over its reliable peers, preventing specialist behavior from being suppressed during teacher aggregation. Experiments on mathematical reasoning, code generation, and instruction following show that the resulting Qwen3-1.7B student consistently outperforms the strongest individual teacher across all three domains, yielding relative improvements of 2.0%, 8.3%, and 6.9%, respectively, while retaining single-model inference. These results establish a simple but powerful principle: stronger students can be obtained not by selecting a single better teacher, but by deliberately constructing and compressing a complementary teacher union.