mlsanityData to 14 Oct 2026

Galvatron: An Automatic Distributed System for Efficient Foundation Model Training

arXiv 2504.21411 ↗First posted 30 Apr 2025cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Galvatron is a distributed system for efficiently training large-scale Foundation Models. It overcomes the complexities of selecting optimal parallelism strategies by automatically identifying the most efficient hybrid strategy, incorporating data, tensor, pipeline, sharded data, and sequence parallelism, along with recomputation. The system's architecture includes a profiler for hardware and model analysis, a search engine for strategy optimization using decision trees and dynamic programming, and a runtime for executing these strategies efficiently. Benchmarking on various clusters demonstrates Galvatron's superior throughput compared to existing frameworks. This open-source system offers user-friendly interfaces and comprehensive documentation, making complex distributed training accessible and efficient. The source code of Galvatron is available at https://github.com/PKU-DAIR/Hetu-Galvatron.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

19 of 25 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
7 more
  1. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  2. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  3. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  4. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  5. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
  6. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  7. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7

Built on by

Not cited yet.