mlsanityData to 14 Oct 2026

PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

arXiv 2606.10369 ↗First posted 9 Jun 2026cs.CL cs.LGcited by 0

On the map: Mixture-of-experts → MoE large language models

also in 1 more
Abstract (arXiv)

As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline. Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 31 of its 38 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

31 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 26

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
14 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  3. 2025Group Sequence Policy Optimizationcited by 835
  4. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  5. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  6. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  8. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  9. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  10. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 72
  11. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  12. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  13. 2025Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generationcited by 16
  14. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Expertscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
  2. 2023DisWOT: Student Architecture Search for Distillation WithOut Training
  3. 2023Can a student Large Language Model perform as well as it's teacher?
  4. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Models
  5. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  6. 2025UNDO: Understanding Distillation as Optimization
  7. 2026MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
  8. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models