# FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention

arXiv: https://arxiv.org/abs/2504.02211 · HTML: https://mlsanity.com/paper/2504.02211/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 58 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [FT-CNN: Algorithm-Based Fault Tolerance for Convolutional Neural Networks](https://mlsanity.com/paper/2003.12203/) (2020)
- [FitAct: Error Resilient Deep Neural Networks via Fine-Grained Post-Trainable Activation Functions](https://mlsanity.com/paper/2112.13544/) (2021)
- [ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training](https://mlsanity.com/paper/2410.11720/) (2024)
- [Soft Error Reliability Analysis of Vision Transformers](https://mlsanity.com/paper/2302.10468/) (2023)
- [Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs](https://mlsanity.com/paper/2104.09455/) (2021)
- [Training Modern Deep Neural Networks for Memory-Fault Robustness](https://mlsanity.com/paper/1911.10287/) (2019)
- [Efficient Error-Tolerant Quantized Neural Network Accelerators](https://mlsanity.com/paper/1912.07394/) (2019)
- [Anatomy of High-Performance GEMM with Online Fault Tolerance on GPUs](https://mlsanity.com/paper/2305.01024/) (2023)
- [TurboFFT: A High-Performance Fast Fourier Transform with Fault Tolerance on GPU](https://mlsanity.com/paper/2405.02520/) (2024)
- [FT K-means: A High-Performance K-means on GPU with Fault Tolerance](https://mlsanity.com/paper/2408.01391/) (2024)
- [Towards Enhancing Fault Tolerance in Neural Networks](https://mlsanity.com/paper/1907.03103/) (2019)

## Built on by (most cited first, 8 of 8)

- [Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference](https://mlsanity.com/paper/2606.02430/) (2026)
- [AttnBoost: Retail Supply Chain Sales Insights via Gradient Boosting Perspective](https://mlsanity.com/paper/2509.10506/) (2025)
- [UTOPIA: Unlearnable Tabular Data via Decoupled Shortcut Embedding](https://mlsanity.com/paper/2602.07358/) (2026)
- [Hierarchical Fault Detection and Diagnosis for Transformer Architectures](https://mlsanity.com/paper/2604.28118/) (2026)
- [CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training](https://mlsanity.com/paper/2605.04478/) (2026)
- [A Foundation Model for Multimodal Event Sequences in Financial Applications](https://mlsanity.com/paper/2607.09955/) (2026)
- [A Cross-lingual Comparison of Human and Classification Model Entrainment Behavior in Code-switched Speech Settings](https://mlsanity.com/paper/2607.25202/) (2026)
- [CheckOne: Lightweight Fault Detection and Mitigation for Vision Transformers](https://mlsanity.com/paper/2608.04035/) (2026)

## Abstract (arXiv)

Transformer models rely on High-Performance Computing (HPC) resources for inference, where soft errors are inevitable in large-scale systems, making the reliability of the model particularly critical. Existing fault tolerance frameworks for Transformers are designed at the operation level without architectural optimization, leading to significant computational and memory overhead, which in turn reduces protection efficiency and limits scalability to larger models. In this paper, we implement module-level protection for Transformers by treating the operations within the attention module as a single kernel and applying end-to-end fault tolerance. This method provides unified protection across multi-step computations, while achieving comprehensive coverage of potential errors in the nonlinear computations. For linear modules, we design a strided algorithm-based fault tolerance (ABFT) that avoids inter-thread communication. Experimental results show that our end-to-end fault tolerance achieves up to 7.56x speedup over traditional methods with an average fault tolerance overhead of 13.9%.