mlsanityData to 7 Oct 2026

Characterization of Large Language Model Development in the Datacenter

arXiv 2403.07648 ↗First posted 12 Mar 2024cs.DC cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs, often riddled with numerous challenges such as frequent hardware failures, intricate parallelization strategies, and imbalanced resource utilization. In this paper, we present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme. Specifically, we investigate discrepancies between LLMs and prior task-specific Deep Learning (DL) workloads, explore resource utilization patterns, and identify the impact of various job failures. Our analysis summarizes hurdles we encountered and uncovers potential opportunities to optimize systems tailored for LLMs. Furthermore, we introduce our system efforts: (1) fault-tolerant pretraining, which enhances fault tolerance through LLM-involved failure diagnosis and automatic recovery. (2) decoupled scheduling for evaluation, which achieves timely performance feedback via trial decomposition and scheduling optimization.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 53 of its 106 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

53 of 106 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2023Mistral 7Bcited by 3,065
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  13. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  14. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  15. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  16. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  17. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  18. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  19. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  20. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  21. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  22. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  23. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  24. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  25. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
  26. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  27. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  28. 2021tf.data: A Machine Learning Data Processing Frameworkcited by 40
  29. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  30. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  31. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  32. 2022Carbon Explorer: A Holistic Approach for Designing Carbon Aware Datacenterscited by 31
  33. 2023Log Parsing with Prompt-based Few-shot Learningcited by 28
  34. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  35. 2023Log Parsing: How Far Can ChatGPT Go?cited by 16
  36. 2022tf.data service: A Case for Disaggregating ML Input Data Processingcited by 15
  37. 2020CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recoverycited by 11
  38. 2024SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validationcited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 18; 2026: 16.

46 papers.

Sort:

By area:

  1. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  2. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  3. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  4. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  5. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 27
  6. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 23
  7. 2025Robust LLM Training Infrastructure at ByteDancecited by 14
  8. 2024Minder: Faulty Machine Detection for Large-scale Distributed Model Trainingcited by 11
  9. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 11
  10. 2024The Unseen AI Disruptions for Power Grids: LLM-Induced Transientscited by 10
  11. 2024Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learningcited by 8
  12. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspectivecited by 8
34 more
  1. 2024A Survey on Failure Analysis and Fault Injection in AI Systemscited by 7
  2. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  3. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 7
  4. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  5. 2025BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inferencecited by 5
  6. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  7. 2025Reliable and Resilient Collective Communication Library for LLM Training and Servingcited by 4
  8. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  9. 2025Green Prompting: Characterizing Prompt-driven Energy Costs of LLM Inferencecited by 3
  10. 2025OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Trainingcited by 3
  11. 2025MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Trainingcited by 3
  12. 2025PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scalingcited by 2
  13. 2025SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Modelscited by 2
  14. 2025Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Servingcited by 2
  15. 2026Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMscited by 2
  16. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2
  17. 2026tLoRA: Efficient Multi-LoRA Training with Elastic Shared Super-Modelscited by 2
  18. 2026HetCCL: Accelerating LLM Training with Heterogeneous GPUscited by 1
  19. 2026Measurement of Generative AI Workload Power Profiles for Whole-Facility Data Center Infrastructure Planningcited by 1
  20. 2026Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Trainingcited by 1
  21. 2026LUMEN: Coordinated Failure Recovery for Distributed LLM Servingcited by 1
  22. 2024Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Designcited by 0
  23. 2024Photon: Federated LLM Pre-Trainingcited by 0
  24. 2025Tesserae: Scalable Placement Policies for Deep Learning Workloadscited by 0
  25. 2025Hybrid Learning and Optimization-Based Dynamic Scheduling for DL Workloads on Heterogeneous GPU Clusterscited by 0
  26. 2026ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workloadcited by 0
  27. 2026Learn-by-Wire Training Control Governance: Bounded Autonomous Training Under Stress for Stability and Efficiencycited by 0
  28. 2026Breaking the Ice: Analyzing Cold Start Latency in vLLMcited by 0
  29. 2026GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failurescited by 0
  30. 2026FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloadscited by 0
  31. 2026Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Servingcited by 0
  32. 2026Characterizing Job Power Elasticity for Power-Flexible AI Trainingcited by 0
  33. 2026MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Servingcited by 0
  34. 2026Leto: Fast In-Place Recovery for LLM Training on Surviving Hardwarecited by 0