# ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training

arXiv: https://arxiv.org/abs/2410.11720 · HTML: https://mlsanity.com/paper/2410.11720/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [Learning Rate Optimization for Federated Learning Exploiting Over-the-air Computation](https://mlsanity.com/paper/2102.02946/) (2021)
- [FT-CNN: Algorithm-Based Fault Tolerance for Convolutional Neural Networks](https://mlsanity.com/paper/2003.12203/) (2020)
- [TRANSOM: An Efficient Fault-Tolerant System for Training LLMs](https://mlsanity.com/paper/2310.10046/) (2023)
- [Detecting silent data corruptions in the wild](https://mlsanity.com/paper/2203.08989/) (2022)
- [Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs](https://mlsanity.com/paper/2007.04457/) (2020)
- [Efficient Soft-Error Detection for Low-precision Deep Learning Recommendation Models](https://mlsanity.com/paper/2103.00130/) (2021)
- [FlipTracker: Understanding Natural Error Resilience in HPC Applications](https://mlsanity.com/paper/1809.01362/) (2018)
- [Improving Energy Saving of One-sided Matrix Decompositions on CPU-GPU Heterogeneous Systems](https://mlsanity.com/paper/2301.03166/) (2023)
- [Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms](https://mlsanity.com/paper/2404.12674/) (2024)

## Built on by (most cited first, 12 of 12)

- [FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention](https://mlsanity.com/paper/2504.02211/) (2025)
- [BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference](https://mlsanity.com/paper/2511.10054/) (2025)
- [Exploring Silent Data Corruption as a Reliability Challenge in LLM Training](https://mlsanity.com/paper/2604.00726/) (2026)
- [Custom Algorithm-based Fault Tolerance for Attention Layers in Transformers](https://mlsanity.com/paper/2507.16676/) (2025)
- [V-ABFT: Variance-Based Adaptive Threshold for Fault-Tolerant Matrix Multiplication in Mixed-Precision Deep Learning](https://mlsanity.com/paper/2602.08043/) (2026)
- [Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.09319/) (2026)
- [All is Not Lost: LLM Recovery without Checkpoints](https://mlsanity.com/paper/2506.15461/) (2025)
- [Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference](https://mlsanity.com/paper/2606.02430/) (2026)
- [Hierarchical Fault Detection and Diagnosis for Transformer Architectures](https://mlsanity.com/paper/2604.28118/) (2026)
- [CheckOne: Lightweight Fault Detection and Mitigation for Vision Transformers](https://mlsanity.com/paper/2608.04035/) (2026)
- [TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training](https://mlsanity.com/paper/2608.30769/) (2026)
- [Syndrome Decoding for Silent Data Corruption in Quantized Integer GPU Arithmetic](https://mlsanity.com/paper/2609.19743/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance in various natural language processing tasks. However, the training of these models is computationally intensive and susceptible to faults, particularly in the attention mechanism, which is a critical component of transformer-based LLMs. In this paper, we investigate the impact of faults on LLM training, focusing on INF, NaN, and near-INF values in the computation results with systematic fault injection experiments. We observe the propagation patterns of these errors, which can trigger non-trainable states in the model and disrupt training, forcing the procedure to load from checkpoints. To mitigate the impact of these faults, we propose ATTNChecker, the first Algorithm-Based Fault Tolerance (ABFT) technique tailored for the attention mechanism in LLMs. ATTNChecker is designed based on fault propagation patterns of LLM and incorporates performance optimization to adapt to both system reliability and model vulnerability while providing lightweight protection for fast LLM training. Evaluations on four LLMs show that ATTNChecker incurs on average 7% overhead on training while detecting and correcting all extreme errors. Compared with the state-of-the-art checkpoint/restore approach, ATTNChecker reduces recovery overhead by up to 49x.