mlsanityData to 7 Oct 2026

ModuleFormer: Modularity Emerges from Mixture-of-Experts

arXiv 2306.04640 ↗First posted 7 Jun 2023cs.CL cs.AI cs.LGcited by 17

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable results. However, existing models are expensive to train and deploy, and it is also difficult to expand their knowledge beyond pre-training data without forgetting previous knowledge. This paper proposes a new neural network architecture, ModuleFormer, that leverages modularity to improve the efficiency and flexibility of large language models. ModuleFormer is based on the Sparse Mixture of Experts (SMoE). Unlike the previous SMoE-based modular language model, which requires domain-labeled data to learn domain-specific experts, ModuleFormer can induce modularity from uncurated data with its new load balancing and concentration losses. ModuleFormer is a modular architecture that includes two different types of modules: new stick-breaking attention heads and feedforward experts. Different modules are sparsely activated conditions on the input token during training and inference. In our experiment, we found that the modular architecture enables three important abilities for large pre-trained language models: 1) Efficiency, since ModuleFormer only activates a subset of its modules for each input token, thus it could achieve the same performance as dense LLMs with more than two times throughput; 2) Extendability, ModuleFormer is more immune to catastrophic forgetting than dense LLMs and can be easily extended with new modules to learn new knowledge that is not included in the training data; 3) Specialisation, finetuning ModuleFormer could specialize a subset of modules to the finetuning task and the task-unrelated modules could be easily pruned for a lightweight deployment.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 52 of its 63 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

52 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
37 more
  1. 2016Learning without Forgettingcited by 2,465
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2019Publicly Available Clinical BERT Embeddingscited by 995
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  13. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  14. 2018Rethinking the Value of Network Pruningcited by 595
  15. 2017Learning multiple visual domains with residual adapterscited by 553
  16. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  17. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  18. 2015Neural Module Networkscited by 375
  19. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  20. 2017Meta Networkscited by 268
  21. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  22. 2021PonderNet: Learning to Pondercited by 156
  23. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  24. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  25. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  26. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  27. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  28. 2022A Length-Extrapolatable Transformercited by 123
  29. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  30. 2020Learning to Continually Learncited by 82
  31. 2019Pruning from Scratchcited by 72
  32. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  33. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  34. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  35. 2019Regularization Shortcomings for Continual Learningcited by 32
  36. 2019IMHO Fine-Tuning Improves Claim Detectioncited by 27
  37. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learnerscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 6; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
  3. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  4. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  5. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  6. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  7. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  8. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12
  9. 2025Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specializationcited by 9
  10. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  11. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Expertscited by 7
  12. 2024Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Expertscited by 7
5 more
  1. 2024Retrieval with Learned Similaritiescited by 3
  2. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Expertscited by 3
  3. 2026EMO: Pretraining Mixture of Experts for Emergent Modularitycited by 3
  4. 2025Neuronal Group Communication for Efficient Neural representationcited by 1
  5. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Unlocking Emergent Modularity in Large Language Models
  2. 2024Scalable Multi-Domain Adaptation of Language Models using Modular Experts
  3. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
  4. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  5. 2025MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
  6. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
  7. 2026NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension
  8. 2025S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning