mlsanityData to 7 Oct 2026

MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation

arXiv 2204.07675 ↗First posted 15 Apr 2022cs.CLcited by 37

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Pre-trained language models have demonstrated superior performance in various natural language processing tasks. However, these models usually contain hundreds of millions of parameters, which limits their practicality because of latency requirements in real-world applications. Existing methods train small compressed models via knowledge distillation. However, performance of these small models drops significantly compared with the pre-trained models due to their reduced model capacity. We propose MoEBERT, which uses a Mixture-of-Experts structure to increase model capacity and inference speed. We initialize MoEBERT by adapting the feed-forward neural networks in a pre-trained model into multiple experts. As such, representation power of the pre-trained model is largely retained. During inference, only one of the experts is activated, such that speed can be improved. We also propose a layer-wise distillation method to train MoEBERT. We validate the efficiency and effectiveness of MoEBERT on natural language understanding and question answering tasks. Results show that the proposed method outperforms existing task-specific distillation algorithms. For example, our method outperforms previous approaches by over 2% on the MNLI (mismatched) dataset. Our code is publicly available at https://github.com/SimiaoZuo/MoEBERT.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 53 references that are in the map. First posted on arXiv 15 Apr 2022.

Builds on

36 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
22 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2018Neural Network Acceptability Judgmentscited by 952
  10. 2019Importance Estimation for Neural Network Pruningcited by 531
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  13. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  14. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  15. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  16. 2021Hash Layers For Large Sparse Modelscited by 236
  17. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  18. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  19. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  20. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  21. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  22. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 13; 2025: 6; 2026: 4.

37 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  3. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  4. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  6. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  7. 2023Language-driven All-in-one Adverse Weather Removalcited by 36
  8. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  9. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 25
  10. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 15
  11. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  12. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12
25 more
  1. 2024CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Informationcited by 11
  2. 2024Multi-student Diffusion Distillation for Better One-step Generatorscited by 9
  3. 2024Flexible and Adaptable Summarization via Expertise Separationcited by 7
  4. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 7
  5. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  6. 20253D Gaussian Splatting Data Compression with Mixture of Priorscited by 5
  7. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  8. 2023Partition-and-Debias: Agnostic Biases Mitigation via A Mixture of Biases-Specific Expertscited by 3
  9. 2023Task-oriented Memory-efficient Pruning-Adaptercited by 2
  10. 2023COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Searchcited by 2
  11. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  12. 2025Mixture of Routerscited by 2
  13. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 2
  14. 2023A Stability Analysis of Fine-Tuning a Pre-Trained Modelcited by 1
  15. 2024Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddingscited by 1
  16. 2024ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillationcited by 1
  17. 2023G-SPEED: General SParse Efficient Editing MoDelcited by 0
  18. 2023On Elastic Language Modelscited by 0
  19. 2024Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Servicescited by 0
  20. 2025NEU-ESC: A Comprehensive Vietnamese dataset for Educational Sentiment analysis and topic Classification toward multitask learningcited by 0
  21. 2025ExpertSim: Fast Particle Detector Simulation Using Mixture-of-Generative-Expertscited by 0
  22. 2026ExFusion: Efficient Transformer Training via Multi-Experts Fusioncited by 0
  23. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  24. 2026A Study on Hidden Layer Distillation for Large Language Model Pre-Trainingcited by 0
  25. 2026KDFP: A first-principles approach to knowledge distillation in large language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  3. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models
  5. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  6. 2024A Survey on Mixture of Experts in Large Language Models
  7. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
  8. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition