mlsanityData to 7 Oct 2026

UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming

arXiv 2307.16375 ↗First posted 31 Jul 2023cs.LG cs.DC math.OCcited by 2

Not on the map. Why?

Abstract (arXiv)

Distributed learning is commonly used for training deep learning models, especially large models. In distributed learning, manual parallelism (MP) methods demand considerable human effort and have limited flexibility. Hence, automatic parallelism (AP) methods have recently been proposed for automating the parallel strategy optimization process. Existing AP methods suffer from sub-optimal solutions because they do not jointly optimize the two categories of parallel strategies (i.e., inter-layer parallelism and intra-layer parallelism). In this paper, we propose a novel AP method called UniAP, which unifies inter- and intra-layer automatic parallelism by mixed integer quadratic programming. To the best of our knowledge, UniAP is the first parallel method that can jointly optimize the two categories of parallel strategies to find an optimal solution. Experimental results show that UniAP outperforms state-of-the-art methods by up to 3.80× in throughput and reduces strategy optimization time by up to 107× across five Transformer-based models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 52 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

33 of 52 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
21 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2017Mixed Precision Trainingcited by 1,135
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  6. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  7. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  8. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  9. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  10. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  11. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  12. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  13. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  14. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  15. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  16. 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
  17. 2018Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networkscited by 23
  18. 2021Automap: Towards Ergonomic Automated Parallelism for ML Modelscited by 12
  19. 2020TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelismcited by 10
  20. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  21. 2020Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloadscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performancecited by 0
  2. 2025Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inferencecited by 0