mlsanityData to 7 Oct 2026

MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

arXiv 2110.01786 ↗First posted 5 Oct 2021cs.CLcited by 149

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain. To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models on various downstream tasks. Besides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs. The source code of this paper can be obtained from https://github.com/thunlp/MoEfication.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 48 of its 67 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

48 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
35 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  13. 2021Editing Factual Knowledge in Language Modelscited by 548
  14. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  15. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  16. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  17. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  18. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  19. 2019Revealing the Dark Secrets of BERTcited by 300
  20. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  21. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  22. 2021Hash Layers For Large Sparse Modelscited by 236
  23. 2013Maxout Networkscited by 186
  24. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  25. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  26. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  27. 2013Deep Learning of Representations: Looking Forwardcited by 84
  28. 2019AllenNLP Interpret: A Framework for Explaining Predictions of NLP Modelscited by 74
  29. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  30. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  31. 2020Finding Experts in Transformer Modelscited by 28
  32. 2020Dynamic Inference: A New Approach Toward Efficient Video Action Recognitioncited by 25
  33. 2020Rethinking the Value of Transformer Componentscited by 24
  34. 2020Know What You Don't Need: Single-Shot Meta-Pruning for Attention Headscited by 16
  35. 2020Towards Interpreting BERT for Reading Comprehension Based QAcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 23; 2024: 55; 2025: 37; 2026: 29.

149 papers.

Sort:

By area:

  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  5. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  6. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  7. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  8. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  9. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  10. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  11. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  12. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
38 more
  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  3. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
  4. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  5. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  6. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  7. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  8. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  9. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
  10. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  11. 2024Dissecting Language Models: Machine Unlearning via Selective Pruningcited by 38
  12. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  13. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  14. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  15. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  16. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  17. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  18. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  19. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 25
  20. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMscited by 24
  21. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  22. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  23. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  24. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
  25. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 19
  26. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 18
  27. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  28. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16
  29. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  30. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 15
  31. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  32. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  33. 2024SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?cited by 14
  34. 2024Memorized Images in Diffusion Models share a Subspace that can be Located and Deletedcited by 13
  35. 2025Adaptive Semantic Token Communication for Transformer-based Edge Inferencecited by 13
  36. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12
  37. 2023Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networkscited by 12
  38. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Secret mixtures of experts inside your LLM
  2. 2022Efficient Sparsely Activated Transformers
  3. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
  4. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformers
  5. 2026What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models
  6. 2024Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers
  7. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
  8. 2025UMoE: Unifying Attention and FFN with Shared Experts