mlsanityData to 14 Oct 2026

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

arXiv 2110.14883 ↗First posted 28 Oct 2021cs.LG cs.AI cs.CLcited by 96

Not on the map. Why?

Abstract (arXiv)

The success of Transformer models has pushed the deep learning model scale to billions of parameters. Due to the limited memory resource of a single GPU, However, the best practice for choosing the optimal parallel strategy is still lacking, since it requires domain expertise in both deep learning and parallel computing. The Colossal-AI system addressed the above challenge by introducing a unified interface to scale your sequential code of model training to distributed environments. It supports parallel training methods such as data, pipeline, tensor, and sequence parallelism, as well as heterogeneous training methods integrated with zero redundancy optimizer. Compared to the baseline system, Colossal-AI can achieve up to 2.76 times training speedup on large-scale models.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Builds on 21 of its 45 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

21 of 45 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
9 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  5. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  6. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  7. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  8. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  9. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 27; 2024: 27; 2025: 24; 2026: 13.

96 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2023Scaling Data-Constrained Language Modelscited by 403
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  6. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 163
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  8. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  9. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  10. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  11. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Frameworkcited by 73
  12. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
38 more
  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 61
  3. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  4. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 57
  5. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  6. 2024A Survey on Transformer Compressioncited by 46
  7. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 44
  8. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  9. 2023A Survey on Efficient Training of Transformerscited by 39
  10. 2024Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applicationscited by 33
  11. 2024Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarkscited by 32
  12. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  13. 2023Reinforcement Learning with Human Feedback for Realistic Traffic Simulationcited by 21
  14. 2024INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuningcited by 20
  15. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  16. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  17. 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
  18. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  19. 2023DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Trainingcited by 15
  20. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  21. 2023TRANSOM: An Efficient Fault-Tolerant System for Training LLMscited by 13
  22. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  23. 2023Blockwise Parallel Transformer for Large Context Modelscited by 12
  24. 2024LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMscited by 12
  25. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  26. 2023GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Lengthcited by 11
  27. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  28. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  29. 2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
  30. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  31. 2023Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabiccited by 7
  32. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  33. 2025Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputerscited by 7
  34. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Schedulingcited by 6
  35. 2023Towards Message Brokers for Generative AI: Survey, Challenges, and Opportunitiescited by 5
  36. 2024Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvementcited by 5
  37. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  38. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5

50 most cited of 96 All 96 (JSON)