mlsanityData to 14 Oct 2026

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

arXiv 2101.03961 ↗First posted 11 Jan 2021cs.LG cs.AIcited by 3,073

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the "Colossal Clean Crawled Corpus" and achieve a 4x speedup over the T5-XXL model.

Key facts. Cited by 3,073 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 42 of its 65 references that are in the map. First posted on arXiv 11 Jan 2021.

Builds on

42 of 65 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  12. 2020Longformer: The Long-Document Transformercited by 3,413
30 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2020Reformer: The Efficient Transformercited by 1,668
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  13. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  14. 2017Mixed Precision Trainingcited by 1,156
  15. 2015Teaching Machines to Read and Comprehendcited by 1,049
  16. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  17. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  18. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  19. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  20. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  21. 2020The Hardware Lotterycited by 160
  22. 2019Adaptively Sparse Transformerscited by 155
  23. 2019Adaptive Attention Span in Transformerscited by 149
  24. 2020Sparse GPU Kernels for Deep Learningcited by 141
  25. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  26. 2019Large Memory Layers with Product Keyscited by 124
  27. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  28. 2018MaskGAN: Better Text Generation via Filling in the______cited by 89
  29. 2020Scalable Transfer Learning with Expert Modelscited by 49
  30. 2014Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 178; 2022: 223; 2023: 314; 2024: 520; 2025: 758; 2026: 1080.

3,073 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
38 more
  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2023A Survey on Multimodal Large Language Modelscited by 909
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  11. 2024DeepSeek-V3 Technical Reportcited by 811
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  13. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  14. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  15. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  16. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  17. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  18. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  19. 2023A Comprehensive Overview of Large Language Modelscited by 648
  20. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  21. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  22. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  23. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  24. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  25. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  26. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  27. 2023The Falcon Series of Open Language Modelscited by 535
  28. 2024Large Language Models: A Surveycited by 525
  29. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  30. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  31. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  32. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  33. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  34. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  35. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  36. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  37. 2021Dynamic Neural Networks: A Surveycited by 408
  38. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407

50 most cited of 3,073 All 3,073 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  2. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  3. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  4. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  5. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
  6. 2024A Survey on Mixture of Experts in Large Language Models
  7. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  8. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization