# Zero Memory Overhead Approach for Protecting Vision Transformer Parameters

arXiv: https://arxiv.org/abs/2507.03816 · HTML: https://mlsanity.com/paper/2507.03816/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 4 Jul 2025.

## Builds on (most cited first, 7 of 7)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [Soft Error Reliability Analysis of Vision Transformers](https://mlsanity.com/paper/2302.10468/) (2023)
- [ALBERTA: ALgorithm-Based Error Resilience in Transformer Architectures](https://mlsanity.com/paper/2310.03841/) (2023)

## Built on by (most cited first, 1 of 1)

- [BitFlipScope: Scalable Fault Localization and Recovery for Bit-Flip Corruptions in LLMs](https://mlsanity.com/paper/2512.22174/) (2025)

## Abstract (arXiv)

Vision Transformers (ViTs) have demonstrated superior performance over Convolutional Neural Networks (CNNs) in various vision-related tasks such as classification, object detection, and segmentation due to their use of self-attention mechanisms. As ViTs become more popular in safety-critical applications like autonomous driving, ensuring their correct functionality becomes essential, especially in the presence of bit-flip faults in their parameters stored in memory. In this paper, a fault tolerance technique is introduced to protect ViT parameters against bit-flip faults with zero memory overhead. Since the least significant bits of parameters are not critical for model accuracy, replacing the LSB with a parity bit provides an error detection mechanism without imposing any overhead on the model. When faults are detected, affected parameters are masked by zeroing out, as most parameters in ViT models are near zero, effectively preventing accuracy degradation. This approach enhances reliability across ViT models, improving the robustness of parameters to bit-flips by up to three orders of magnitude, making it an effective zero-overhead solution for fault tolerance in critical applications.