mlsanityData to 7 Oct 2026

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

arXiv 2006.16668 ↗First posted 30 Jun 2020cs.CL cs.LG stat.MLcited by 1,601

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Neural network scaling has been critical for improving the model quality in many real-world machine learning applications with vast amounts of training data and compute. Although this trend of scaling is affirmed to be a sure-fire approach for better model quality, there are challenges on the path such as the computation cost, ease of programming, and efficient implementation on parallel devices. GShard is a module composed of a set of lightweight annotation APIs and an extension to the XLA compiler. It provides an elegant way to express a wide range of parallel computation patterns with minimal changes to the existing model code. GShard enabled us to scale up multilingual neural machine translation Transformer model with Sparsely-Gated Mixture-of-Experts beyond 600 billion parameters using automatic sharding. We demonstrate that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art.

Key facts. Cited by 1,601 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 57 of its 99 references that are in the map. First posted on arXiv 30 Jun 2020.

Builds on

57 of 99 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
38 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  11. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  12. 2017Exploring Generalization in Deep Learningcited by 595
  13. 2015Listen, Attend and Spellcited by 542
  14. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  15. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  16. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  17. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 289
  18. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  19. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  20. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  21. 2019Massively Multilingual Neural Machine Translationcited by 246
  22. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  23. 2019Depth-Adaptive Transformercited by 227
  24. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  25. 2019The Evolved Transformercited by 180
  26. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  27. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  28. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  29. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  30. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  31. 2019Language Modeling with Deep Transformerscited by 96
  32. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  33. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  34. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  35. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  36. 2018Glow: Graph Lowering Compiler Techniques for Neural Networkscited by 73
  37. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  38. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 98; 2022: 113; 2023: 155; 2024: 309; 2025: 405; 2026: 520.

1,601 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2024Mixtral of Expertscited by 1,785
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
38 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2021Scaling Vision Transformerscited by 993
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  8. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2025Kimi K2: Open Agentic Intelligencecited by 523
  13. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  15. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  16. 2021A Survey of Transformerscited by 399
  17. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  18. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  19. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  20. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  21. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  22. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  23. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  24. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  25. 2022Confident Adaptive Language Modelingcited by 297
  26. 2022UL2: Unifying Language Learning Paradigmscited by 292
  27. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  28. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  29. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  30. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  31. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  32. 2023From Sparse to Soft Mixtures of Expertscited by 245
  33. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  34. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  35. 2021Hash Layers For Large Sparse Modelscited by 236
  36. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  37. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  38. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228

50 most cited of 1,601 All 1,601 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Models
  2. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
  3. 2025Pruning Laws for Large Language Models
  4. 2024Lillama: Large Language Models Compression via Low-Rank Feature Distillation
  5. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  6. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  7. 2025From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
  8. 2024Search for Efficient Large Language Models