mlsanityData to 14 Oct 2026

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

arXiv 2211.13878 ↗First posted 25 Nov 2022cs.LG cs.DB cs.DCcited by 53

Not on the map. Why?

Abstract (arXiv)

Transformer models have achieved state-of-the-art performance on various domains of applications and gradually becomes the foundations of the advanced large deep learning (DL) models. However, how to train these models over multiple GPUs efficiently is still challenging due to a large number of parallelism choices. Existing DL systems either rely on manual efforts to make distributed training plans or apply parallelism combinations within a very limited search space. In this approach, we propose Galvatron, a new system framework that incorporates multiple popular parallelism dimensions and automatically finds the most efficient hybrid parallelism strategy. To better explore such a rarely huge search space, we 1) involve a decision tree to make decomposition and pruning based on some reasonable intuitions, and then 2) design a dynamic programming search algorithm to generate the optimal plan. Evaluations on four representative Transformer workloads show that Galvatron could perform automatically distributed training with different GPU memory budgets. Among all evluated scenarios, Galvatron always achieves superior system throughput compared to previous work with limited parallelism.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

32 of 52 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
20 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  2. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  5. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  6. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  7. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  8. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  9. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  10. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  11. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 55
  12. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  13. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training Systemcited by 36
  14. 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
  15. 2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
  16. 2020Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platformscited by 25
  17. 2018Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networkscited by 23
  18. 2022OSDP: Optimal Sharded Data Parallel for Distributed Deep Learningcited by 12
  19. 2018GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelismcited by 10
  20. 2020TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelismcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 11; 2025: 15; 2026: 18.

53 papers.

Sort:

By area:

  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  3. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  4. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  5. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  6. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 33
  7. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 31
  8. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  9. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  10. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  11. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 14
  12. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
38 more
  1. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  2. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  3. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  4. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  5. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Servingcited by 9
  6. 2023Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencentcited by 8
  7. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 8
  8. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7
  9. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  10. 2025Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Trainingcited by 7
  11. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  12. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  13. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 6
  14. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5
  15. 2024MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Trainingcited by 5
  16. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  17. 2024ProTrain: Efficient LLM Training via Memory-Aware Techniquescited by 4
  18. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  19. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  20. 2025LoRAFusion: Efficient LoRA Fine-Tuning for LLMscited by 3
  21. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 3
  22. 2023UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programmingcited by 2
  23. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  24. 2026How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Servingcited by 2
  25. 2025iServe: An Intent-based Serving System for LLMscited by 1
  26. 2026AutoScout: Structured Optimization for Automating ML System Configurationcited by 1
  27. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  28. 2026FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Trainingcited by 1
  29. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 1
  30. 2025Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approachcited by 0
  31. 2025Galvatron: An Automatic Distributed System for Efficient Foundation Model Trainingcited by 0
  32. 2025Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelizationcited by 0
  33. 2025ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniquescited by 0
  34. 2026Placement Semantics for Distributed Deep Learning: A Systematic Framework for Analyzing Parallelism Strategiescited by 0
  35. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  36. 2026FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Modelscited by 0
  37. 2026Parallelism Strategy Chaining for Fast Training Convergencecited by 0
  38. 2026The Life of a Token: from Words to Bits on the Wirecited by 0

50 most cited of 53 All 53 (JSON)