mlsanityData to 7 Oct 2026

Mesh-TensorFlow: Deep Learning for Supercomputers

arXiv 1811.02084 ↗First posted 5 Nov 2018cs.LG cs.DC stat.MLcited by 168

Not on the map. Why?

Cited by 33 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Batch-splitting (data-parallelism) is the dominant distributed Deep Neural Network (DNN) training strategy, due to its universal applicability and its amenability to Single-Program-Multiple-Data (SPMD) programming. However, batch-splitting suffers from problems including the inability to train very large models (due to memory constraints), high latency, and inefficiency at small batch sizes. All of these can be solved by more general distribution strategies (model-parallelism). Unfortunately, efficient model-parallel algorithms tend to be complicated to discover, describe, and to implement, particularly on large clusters. We introduce Mesh-TensorFlow, a language for specifying a general class of distributed tensor computations. Where data-parallelism can be viewed as splitting tensors and operations along the "batch" dimension, in Mesh-TensorFlow, the user can specify any tensor-dimensions to be split across any dimensions of a multi-dimensional mesh of processors. A Mesh-TensorFlow graph compiles into a SPMD program consisting of parallel operations coupled with collective communication primitives such as Allreduce. We use Mesh-TensorFlow to implement an efficient data-parallel, model-parallel version of the Transformer sequence-to-sequence model. Using TPU meshes of up to 512 cores, we train Transformer models with up to 5 billion parameters, surpassing state of the art results on WMT'14 English-to-French translation task and the one-billion-word language modeling benchmark. Mesh-Tensorflow is available at https://github.com/tensorflow/mesh .

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 52; 2022: 36; 2023: 28; 2024: 22; 2025: 21; 2026: 8.

168 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  8. 2023The Falcon Series of Open Language Modelscited by 526
  9. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  10. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 356
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
38 more
  1. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  2. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  3. 2021Accelerating Sparse Deep Neural Networkscited by 213
  4. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  5. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  6. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  7. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  8. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  9. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  10. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  11. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  12. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  13. 2021Scaling Up Influence Functionscited by 130
  14. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  15. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  16. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  17. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  18. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  19. 2021The Efficiency Misnomercited by 98
  20. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  21. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  22. 2021Sparse is Enough in Scaling Transformerscited by 86
  23. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  24. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  25. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  26. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  27. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  28. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  29. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  30. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  31. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  32. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  33. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  34. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  35. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  36. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  37. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  38. 2021Joint Passage Ranking for Diverse Multi-Answer Retrievalcited by 43

50 most cited of 168 All 168 (JSON)