mlsanityData to 7 Oct 2026

BASE Layers: Simplifying Training of Large, Sparse Models

arXiv 2103.16716 ↗First posted 30 Mar 2021cs.CLcited by 315

Not on the map. Why?

Cited by 231 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new balanced assignment of experts (BASE) layer for large language models that greatly simplifies existing high capacity sparse layers. Sparse layers can dramatically improve the efficiency of training and inference by routing each token to specialized expert modules that contain only a small fraction of the model parameters. However, it can be difficult to learn balanced routing functions that make full use of the available experts; existing approaches typically use routing heuristics or auxiliary expert-balancing loss functions. In contrast, we formulate token-to-expert allocation as a linear assignment problem, allowing an optimal assignment in which each expert receives an equal number of tokens. This optimal assignment scheme improves efficiency by guaranteeing balanced compute loads, and also simplifies training by not requiring any new hyperparameters or auxiliary losses. Code is publicly released at https://github.com/pytorch/fairseq/

Key facts. Cited by 315 papers in the map (data through 7 Oct 2026). Builds on 28 of its 32 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

28 of 32 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2016Layer Normalizationcited by 5,579
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
16 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  9. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  11. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  12. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  13. 2020Nearest Neighbor Machine Translationcited by 209
  14. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  15. 2019Adaptively Sparse Transformerscited by 154
  16. 2019Large Memory Layers with Product Keyscited by 120

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 47; 2023: 51; 2024: 80; 2025: 57; 2026: 62.

315 papers.

Sort:

By area:

  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  8. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  9. 2023From Sparse to Soft Mixtures of Expertscited by 245
  10. 2021Hash Layers For Large Sparse Modelscited by 236
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
38 more
  1. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  2. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  3. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  6. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  9. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  10. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  11. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  12. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  13. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  14. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  15. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  16. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  17. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  18. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  19. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  20. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  21. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  22. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  23. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  24. 2026Kimi K3: Open Frontier Intelligencecited by 99
  25. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  26. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  27. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  28. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  29. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  30. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  31. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  32. 2023Modular Deep Learningcited by 82
  33. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  34. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  35. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  36. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  37. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  38. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69

50 most cited of 315 All 315 (JSON)