# Revisiting Reliability in Large-Scale Machine Learning Research Clusters

arXiv: https://arxiv.org/abs/2410.21680 · HTML: https://mlsanity.com/paper/2410.21680/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 74 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022)
- [Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack](https://mlsanity.com/paper/2309.15807/) (2023)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads](https://mlsanity.com/paper/1901.05758/) (2019)
- [Characterization of Large Language Model Development in the Datacenter](https://mlsanity.com/paper/2403.07648/) (2024)
- [Silent Data Corruptions at Scale](https://mlsanity.com/paper/2102.11245/) (2021)
- [Unicron: Economizing Self-Healing LLM Training at Scale](https://mlsanity.com/paper/2401.00134/) (2023)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters](https://mlsanity.com/paper/2307.12169/) (2023)
- [CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery](https://mlsanity.com/paper/2011.02999/) (2020)
- [The infrastructure powering IBM's Gen AI model development](https://mlsanity.com/paper/2407.05467/) (2024)

## Built on by (most cited first, 28 of 28)

- [AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench](https://mlsanity.com/paper/2507.02554/) (2025)
- [Robust LLM Training Infrastructure at ByteDance](https://mlsanity.com/paper/2509.16293/) (2025)
- [BTS: Harmonizing Specialized Experts into a Generalist LLM](https://mlsanity.com/paper/2502.00075/) (2025)
- [Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training](https://mlsanity.com/paper/2509.03018/) (2025)
- [Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs](https://mlsanity.com/paper/2503.11901/) (2025)
- [Reliable and Resilient Collective Communication Library for LLM Training and Serving](https://mlsanity.com/paper/2512.25059/) (2025)
- [Sparse Checkpointing for Fast and Reliable MoE Training](https://mlsanity.com/paper/2412.15411/) (2024)
- [Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs](https://mlsanity.com/paper/2601.13655/) (2026)
- [GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving](https://mlsanity.com/paper/2605.00831/) (2026)
- [FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference](https://mlsanity.com/paper/2607.10186/) (2026)
- [TrainMover: An Interruption-Resilient Runtime for ML Training](https://mlsanity.com/paper/2412.12636/) (2024)
- [xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloads](https://mlsanity.com/paper/2510.21048/) (2025)
- [Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training](https://mlsanity.com/paper/2605.17879/) (2026)
- [LUMEN: Coordinated Failure Recovery for Distributed LLM Serving](https://mlsanity.com/paper/2606.17787/) (2026)
- [Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training](https://mlsanity.com/paper/2504.06095/) (2025)
- [Evolving HPC services to enable ML workloads on HPE Cray EX](https://mlsanity.com/paper/2507.01880/) (2025)
- [Kant: An Efficient Unified Scheduling System for Large-Scale AI Clusters](https://mlsanity.com/paper/2510.01256/) (2025)
- [Crash-Consistent Checkpointing for AI Training on macOS/APFS](https://mlsanity.com/paper/2511.18323/) (2025)
- [Photons = Tokens: The Physics of AI and the Economics of Knowledge](https://mlsanity.com/paper/2603.06630/) (2026)
- [From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs](https://mlsanity.com/paper/2605.09370/) (2026)
- [ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload](https://mlsanity.com/paper/2605.11215/) (2026)
- [Orbax: Distributed Checkpointing with JAX](https://mlsanity.com/paper/2605.23066/) (2026)
- [ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training](https://mlsanity.com/paper/2605.24326/) (2026)
- [Don't Let a Few Network Failures Slow the Entire AllReduce](https://mlsanity.com/paper/2606.01680/) (2026)
- [GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures](https://mlsanity.com/paper/2607.01409/) (2026)
- [DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on 60 GPUs](https://mlsanity.com/paper/2608.14614/) (2026)
- [Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware](https://mlsanity.com/paper/2610.00687/) (2026)
- [Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines](https://mlsanity.com/paper/2610.03087/) (2026)

## Abstract (arXiv)

Reliability is a fundamental challenge in operating large-scale machine learning (ML) infrastructures, particularly as the scale of ML models and training clusters continues to grow. Despite decades of research on infrastructure failures, the impact of job failures across different scales remains unclear. This paper presents a view of managing two large, multi-tenant ML clusters, providing quantitative analysis, operational experience, and our own perspective in understanding and addressing reliability concerns at scale. Our analysis reveals that while large jobs are most vulnerable to failures, smaller jobs make up the majority of jobs in the clusters and should be incorporated into optimization objectives. We identify key workload properties, compare them across clusters, and demonstrate essential reliability requirements for pushing the boundaries of ML training at scale. We hereby introduce a taxonomy of failures and key reliability metrics, analyze 11 months of data from two state-of-the-art ML environments with 4 million jobs and over 150 million A100 GPU hours. Building on our data, we fit a failure model to project Mean Time to Failure for various GPU scales. We further propose a method to estimate a related metric, Effective Training Time Ratio, as a function of job parameters, and we use this model to gauge the efficacy of potential software mitigations at scale. Our work provides valuable insights and future research directions for improving the reliability of AI supercomputer clusters, emphasizing the need for flexible, workload-agnostic, and reliability-aware infrastructure, system software, and algorithms.