# Silent Data Corruptions at Scale

arXiv: https://arxiv.org/abs/2102.11245 · HTML: https://mlsanity.com/paper/2102.11245/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2021.

## Built on by (most cited first, 23 of 23)

- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Kwai Keye-VL 1.5 Technical Report](https://mlsanity.com/paper/2509.01563/) (2025)
- [Kwai Keye-VL Technical Report](https://mlsanity.com/paper/2507.01949/) (2025)
- [Revisiting Reliability in Large-Scale Machine Learning Research Clusters](https://mlsanity.com/paper/2410.21680/) (2024)
- [Robust LLM Training Infrastructure at ByteDance](https://mlsanity.com/paper/2509.16293/) (2025)
- [Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs](https://mlsanity.com/paper/2104.09455/) (2021)
- [Towards a Safety Case for Hardware Fault Tolerance in Convolutional Neural Networks Using Activation Range Supervision](https://mlsanity.com/paper/2108.07019/) (2021)
- [ReaLM: Reliable and Efficient Large Language Model Inference with Statistical Algorithm-Based Fault Tolerance](https://mlsanity.com/paper/2503.24053/) (2025)
- [Efficient Soft-Error Detection for Low-precision Deep Learning Recommendation Models](https://mlsanity.com/paper/2103.00130/) (2021)
- [Towards a future space-based, highly scalable AI infrastructure system design](https://mlsanity.com/paper/2511.19468/) (2025)
- [Exploring Silent Data Corruption as a Reliability Challenge in LLM Training](https://mlsanity.com/paper/2604.00726/) (2026)
- [FT K-means: A High-Performance K-means on GPU with Fault Tolerance](https://mlsanity.com/paper/2408.01391/) (2024)
- [Harden Deep Neural Networks Against Fault Injections Through Weight Scaling](https://mlsanity.com/paper/2411.18993/) (2024)
- [Effective and Memory-Efficient Alternatives to ECC for Reliable Large-Scale DNNs](https://mlsanity.com/paper/2605.07417/) (2026)
- [Zero knowledge verification for frontier AI training is possible](https://mlsanity.com/paper/2606.05433/) (2026)
- [Hardware Resilience Properties of Text-Guided Image Classifiers](https://mlsanity.com/paper/2311.14062/) (2023)
- [Beyond Moore's Law: Harnessing the Redshift of Generative AI with Effective Hardware-Software Co-Design](https://mlsanity.com/paper/2504.06531/) (2025)
- [FLARE: One-Shot PE-Level Fault Localization in Systolic Arrays via Algebraic Test Vectors](https://mlsanity.com/paper/2605.08594/) (2026)
- [Enhancing Neural Network Robustness Against Fault Injection Through Non-linear Weight Transformations](https://mlsanity.com/paper/2411.19027/) (2024)
- [SafeCiM: Investigating Resilience of Hybrid Floating-Point Compute-in-Memory Deep Learning Accelerators](https://mlsanity.com/paper/2512.00059/) (2025)
- [Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Models](https://mlsanity.com/paper/2607.10389/) (2026)
- [TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training](https://mlsanity.com/paper/2608.30769/) (2026)
- [Silent Failures Beyond the 32-Bit Index Range: A Differential Characterization of Large-Tensor Matrix Multiplication in PyTorch's MPS Backend](https://mlsanity.com/paper/2609.22991/) (2026)

## Abstract (arXiv)

Silent Data Corruption (SDC) can have negative impact on large-scale infrastructure services. SDCs are not captured by error reporting mechanisms within a Central Processing Unit (CPU) and hence are not traceable at the hardware level. However, the data corruptions propagate across the stack and manifest as application-level problems. These types of errors can result in data loss and can require months of debug engineering time. In this paper, we describe common defect types observed in silicon manufacturing that leads to SDCs. We discuss a real-world example of silent data corruption within a datacenter application. We provide the debug flow followed to root-cause and triage faulty instructions within a CPU using a case study, as an illustration on how to debug this class of errors. We provide a high-level overview of the mitigations to reduce the risk of silent data corruptions within a large production fleet. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across generations. We have monitored SDCs for a period longer than 18 months. Based on this experience, we determine that reducing silent data corruptions requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.