mlsanityData to 7 Oct 2026

One Student Knows All Experts Know: From Sparse to Dense

arXiv 2201.10890 ↗First posted 26 Jan 2022cs.LG cs.AI cs.CLcited by 25

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Human education system trains one student by multiple experts. Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts. However, sparse MoE model is easy to overfit, hard to deploy, and not hardware-friendly for practitioners. In this work, inspired by the human education model, we propose a novel task, knowledge integration, to obtain a dense student model (OneS) as knowledgeable as one sparse MoE. We investigate this task by proposing a general training framework including knowledge gathering and knowledge distillation. Specifically, to gather key knowledge from different pre-trained experts, we first investigate four different possible knowledge gathering methods, ie summation, averaging, Top-K Knowledge Gathering (Top-KG), and Singular Value Decomposition Knowledge Gathering (SVD-KG) proposed in this paper. We then refine the dense student model by knowledge distillation to offset the noise from gathering. On ImageNet, our OneS preserves 61.7% benefits from MoE and achieves 78.4% top-1 accuracy ImageNet with only 15M parameters. On four natural language processing datasets, OneS obtains 88.2% MoE benefits and outperforms the best baseline by 51.7% using the same architecture and training data. In addition, compared with the MoE counterpart, OneS can achieve 3.7 × inference speedup due to less computation and hardware-friendly architecture.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 26 of its 37 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

26 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Go Wider Instead of Deeper25 Jul
  2. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
12 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  9. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  10. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  11. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  12. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 6; 2025: 6; 2026: 4.

25 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  3. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  4. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  6. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  7. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoderscited by 25
  8. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  9. 2025Mixture of Experts in Large Language Modelscited by 15
  10. 2022MoEC: Mixture of Expert Clusterscited by 14
  11. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  12. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
13 more
  1. 2024XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Expertscited by 7
  2. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  3. 2023Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulationcited by 5
  4. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  5. 2022A Study on Transformer Configuration and Training Objectivecited by 4
  6. 2024Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Modelscited by 4
  7. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  8. 2023One-stop Training of Multiple Capacity Modelscited by 0
  9. 2024Unity in Diversity: Multi-expert Knowledge Confrontation and Collaboration for Generalizable Vehicle Re-identificationcited by 0
  10. 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hackingcited by 0
  11. 2026Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddingscited by 0
  12. 2026Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMscited by 0
  13. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  2. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  3. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  4. 2024MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts
  5. 2025Sigma-MoE-Tiny Technical Report
  6. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
  7. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  8. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts