mlsanityData to 14 Oct 2026

TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks

arXiv 2302.00247 ↗First posted 1 Feb 2023cs.LG cs.AI cs.DCcited by 3

Not on the map. Why?

Abstract (arXiv)

Tensor parallelism is an essential technique for distributed training of large neural networks. However, automatically determining an optimal tensor parallel strategy is challenging due to the gigantic search space, which grows exponentially with model size and tensor dimension. This prohibits the adoption of auto-parallel systems on larger models. We observe that neural networks usually contain repeated substructures, and build an automatic parallelism framework named TAPAS that eliminates redundant search efforts. TAPAS employs a divide-and-conquer approach that efficiently folds the search space by identifying those unique substructures. As a result, it runs at sub-linear complexity concerning the model size, making it a scalable solution for training large-scale networks. Our evaluations demonstrate that TAPAS outperforms the state-of-the-art automatic parallelism frameworks by up to 160× in search speed on a wide range of models, and the performance of derived strategies is competitive or even better compared with the expert-engineered Megatron-LM library.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 58 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

22 of 58 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2017Mixed Precision Trainingcited by 1,156
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 848
10 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  2. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  3. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  4. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  5. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  6. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  7. 2021Go Wider Instead of Deepercited by 57
  8. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  9. 2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
  10. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.

3 papers.

  1. 2026Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Trainingcited by 1
  2. 2023ROAM: memory-efficient large DNN training via optimized operator ordering and memory layoutcited by 0
  3. 2025Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inferencecited by 0