From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
Not on the map. Why?
Abstract (arXiv)
Large-scale AI training is fundamentally a distributed systems problem, where hardware failures are routine operating conditions rather than rare exceptions, yet public operational evidence from production training clusters remains limited. This report presents an empirical analysis of a 63-node NVIDIA B200 production cluster (504 GPUs), using 55 days of Prometheus time-series data and 73 days of operational logs covering 224 multi-node training sessions. The environment is cross-organizational: five parties (SKT, Upstage, Lablup, NVIDIA Korea, VAST Data) share a unified monitoring pipeline. This enabled joint diagnosis of a 60-node-scale storage I/O bottleneck absent in 2-4-node tests, a production-scale phenomenon no single team could isolate alone. We perform three quantitative analyses yielding four findings. First, over 751 Prometheus metrics and 10 XID-identified GPU failures, no single metric is consistently dominant across failure types, motivating multi-signal detection. Second, 523 checkpoint events trace the save/load path from GPU VRAM to the NFS server: restart loading reaches 21.5% of maximum read bandwidth (700 GB/s) and save bursts 16.0% of maximum write bandwidth (250 GB/s), with NFS/RPC queueing and transport-layer backlog rising together. Third, across 224 sessions over 73 days, node exclusions concentrate so the top 3 of 63 nodes account for over 50%. Fourth, auto-retry chain analysis shows a 33.3% success rate over 12 chains (73 attempts), 2.7x the 12.5% manual rate, with a median retry interval of 11 minutes (IQR 10-11). All analyses are grounded in production infrastructure providing session-level workload management, GPU-centric scheduling, and unified observability.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 49 references that are in the map. First posted on arXiv 10 May 2026.
Builds on
21 of 49 references.
References · 21
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
- 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
- 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 68
- 2024Scaling FP8 training to trillion-token LLMscited by 63
- 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
- 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
- 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
9 more
- 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 27
- 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
- 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 23
- 2023TRANSOM: An Efficient Fault-Tolerant System for Training LLMscited by 12
- 2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
- 2022Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learningcited by 11
- 2024Minder: Faulty Machine Detection for Large-scale Distributed Model Trainingcited by 11
- 2024Lazarus: Resilient and Elastic Training of Mixture-of-Experts Modelscited by 9
- 2024FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Trainingcited by 4
Built on by
Not cited yet.