mlsanityData to 7 Oct 2026

Training and Serving System of Foundation Models: A Comprehensive Survey

arXiv 2401.02643 ↗First posted 5 Jan 2024cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Foundation models (e.g., ChatGPT, DALL-E, PengCheng Mind, PanGu-Sigma) have demonstrated extraordinary performance in key technological areas, such as natural language processing and visual recognition, and have become the mainstream trend of artificial general intelligence. This has led more and more major technology giants to dedicate significant human and financial resources to actively develop their foundation model systems, which drives continuous growth of these models' parameters. As a result, the training and serving of these models have posed significant challenges, including substantial computing power, memory consumption, bandwidth demands, etc. Therefore, employing efficient training and serving strategies becomes particularly crucial. Many researchers have actively explored and proposed effective methods. So, a comprehensive survey of them is essential for system developers and researchers. This paper extensively explores the methods employed in training and serving foundation models from various perspectives. It provides a detailed categorization of these state-of-the-art methods, including finer aspects such as network, computing, and storage. Additionally, the paper summarizes the challenges and presents a perspective on the future development direction of foundation model systems. Through comprehensive discussion and analysis, it hopes to provide a solid theoretical basis and practical guidance for future research and applications, promoting continuous innovation and development in foundation model systems.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 58 of its 103 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

58 of 103 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2017Mixed Precision Trainingcited by 1,135
38 more
  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  5. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  10. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  12. 2023Explainability for Large Language Models: A Surveycited by 331
  13. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  14. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  15. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  16. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  17. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  18. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  19. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  20. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  21. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  22. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  23. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  24. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  25. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  26. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  27. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  28. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  29. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 50
  30. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  31. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  32. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  33. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  34. 2020The Adapter-Bot: All-In-One Controllable Conversational Modelcited by 38
  35. 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
  36. 2021Maximizing Parallelism in Distributed Training for Huge Neural Networkscited by 23
  37. 2023Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiencycited by 20
  38. 2021Tesseract: Parallelize the Tensor Parallelism Efficientlycited by 19

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  3. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  4. 2025Virtual Cells: Predict, Explain, Discovercited by 7
  5. 2025Robustness Tokens: Towards Adversarial Robustness of Transformerscited by 2
  6. 2025Zero-Trust Foundation Models: A New Paradigm for Secure and Collaborative Artificial Intelligence for Internet of Thingscited by 1
  7. 2025Generative AI as a Pillar for Predicting 2D and 3D Wildfire Spread: Beyond Physics-Based Models and Traditional Deep Learningcited by 1
  8. 2026Odyssey: Constructing Verifiable Local Truth-Preserving Foundation Modelscited by 1
  9. 2024ALow-Cost Real-Time Framework for Industrial Action Recognition Using Foundation Modelscited by 0
  10. 2026Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placementcited by 0
  11. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0