# Understanding Silent Data Corruption in LLM Training arXiv: https://arxiv.org/abs/2502.12340 ยท HTML: https://mlsanity.com/paper/2502.12340/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 58 references that are in the map. - First posted on arXiv 17 Feb 2025. ## Builds on (most cited first, 22 of 22) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021) - [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022) - [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019) - [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018) - [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022) - [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019) - [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020) - [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023) - [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019) - [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017) - [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017) - [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016) - [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019) - [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021) - [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022) - [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019) - [Sharpness-Aware Minimization Improves Language Model Generalization](https://mlsanity.com/paper/2110.08529/) (2021) - [Analyzing and Mitigating the Impact of Permanent Faults on a Systolic Array Based Neural Network Accelerator](https://mlsanity.com/paper/1802.04657/) (2018) - [Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs](https://mlsanity.com/paper/2104.09455/) (2021) - [Evaluating and Enhancing Robustness of Deep Recommendation Systems Against Hardware Errors](https://mlsanity.com/paper/2307.10244/) (2023) ## Built on by (most cited first, 15 of 15) - [Robust LLM Training Infrastructure at ByteDance](https://mlsanity.com/paper/2509.16293/) (2025) - [Characterization and Mitigation of Training Instabilities in Microscaling Formats](https://mlsanity.com/paper/2506.20752/) (2025) - [DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responses](https://mlsanity.com/paper/2512.02282/) (2025) - [Tracing the Data Trail: A Survey of Data Provenance, Transparency and Traceability in LLMs](https://mlsanity.com/paper/2601.14311/) (2026) - [Exploring Silent Data Corruption as a Reliability Challenge in LLM Training](https://mlsanity.com/paper/2604.00726/) (2026) - [Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Silicon](https://mlsanity.com/paper/2604.22032/) (2026) - [GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving](https://mlsanity.com/paper/2605.00831/) (2026) - [AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training](https://mlsanity.com/paper/2601.11864/) (2026) - [Analysis of LLM Vulnerability to GPU Soft Errors: An Instruction-Level Fault Injection Study](https://mlsanity.com/paper/2601.19912/) (2025) - [Silent Inconsistency in Data-Parallel Full Fine-Tuning: Diagnosing Worker-Level Optimization Misalignment](https://mlsanity.com/paper/2602.14462/) (2026) - [SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training](https://mlsanity.com/paper/2608.11034/) (2026) - [TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training](https://mlsanity.com/paper/2608.30769/) (2026) - [Syndrome Decoding for Silent Data Corruption in Quantized Integer GPU Arithmetic](https://mlsanity.com/paper/2609.19743/) (2026) - [Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware](https://mlsanity.com/paper/2610.00687/) (2026) - [Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines](https://mlsanity.com/paper/2610.03087/) (2026) ## Abstract (arXiv) As the scale of training large language models (LLMs) increases, one emergent failure is silent data corruption (SDC), where hardware produces incorrect computations without explicit failure signals. In this work, we are the first to investigate the impact of real-world SDCs on LLM training by comparing model training between healthy production nodes and unhealthy nodes exhibiting SDCs. With the help from a cloud computing platform, we access the unhealthy nodes that were swept out from production by automated fleet management. Using deterministic execution via XLA compiler and our proposed synchronization mechanisms, we isolate and analyze the impact of SDC errors on these nodes at three levels: at each submodule computation, at a single optimizer step, and at a training period. Our results reveal that the impact of SDCs on computation varies on different unhealthy nodes. Although in most cases the perturbations from SDCs on submodule computation and gradients are relatively small, SDCs can lead models to converge to different optima with different weights and even cause spikes in the training loss. Our analysis sheds light on further understanding and mitigating the impact of SDCs.