# Statistical Test for Attention Map in Vision Transformer

arXiv: https://arxiv.org/abs/2401.08169 · HTML: https://mlsanity.com/paper/2401.08169/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Exact post-selection inference, with application to the lasso](https://mlsanity.com/paper/1311.6238/) (2013)
- [TransMed: Transformers Advance Multi-modal Medical Image Classification](https://mlsanity.com/paper/2103.05940/) (2021)
- [Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm](https://mlsanity.com/paper/1906.03720/) (2019)
- [Optimal Inference After Model Selection](https://mlsanity.com/paper/1410.2597/) (2014)
- [Exact Post-Selection Inference for Sequential Regression Procedures](https://mlsanity.com/paper/1401.3889/) (2014)
- [Selective Inference for Hierarchical Clustering](https://mlsanity.com/paper/2012.02936/) (2020)
- [Valid Inference Corrected for Outlier Removal](https://mlsanity.com/paper/1711.10635/) (2017)
- [Computing Valid p-value for Optimal Changepoint by Selective Inference using Dynamic Programming](https://mlsanity.com/paper/2002.09132/) (2020)
- [More powerful post-selection inference, with application to the Lasso](https://mlsanity.com/paper/1801.09037/) (2018)
- [Quantifying Statistical Significance of Neural Network-based Image Segmentation by Selective Inference](https://mlsanity.com/paper/2010.01823/) (2020)
- [Selecting the number of principal components: estimation of the true rank of a noisy matrix](https://mlsanity.com/paper/1410.8260/) (2014)
- [A significance test for forward stepwise model selection](https://mlsanity.com/paper/1405.3920/) (2014)
- [Selective Sequential Model Selection](https://mlsanity.com/paper/1512.02565/) (2015)
- [Valid P-Value for Deep Learning-Driven Salient Region](https://mlsanity.com/paper/2301.02437/) (2023)

## Built on by (most cited first, 6 of 6)

- [Statistical Test for Anomaly Detections by Variational Auto-Encoders](https://mlsanity.com/paper/2402.03724/) (2024)
- [Controllable RANSAC-based Anomaly Detection via Hypothesis Testing](https://mlsanity.com/paper/2410.15133/) (2024)
- [Statistical Test for Diffusion-Based Anomaly Localization via Selective Inference](https://mlsanity.com/paper/2402.11789/) (2024)
- [Statistical Inference for Clustering-based Anomaly Detection](https://mlsanity.com/paper/2504.18633/) (2025)
- [Automatic Statistical Test for Rationally Expressible Algorithms by Selective Inference, with Applications to Feature Selection](https://mlsanity.com/paper/2608.04667/) (2026)
- [Post-Anomaly Detection Inference for Deep SVDD](https://mlsanity.com/paper/2609.37935/) (2026)

## Abstract (arXiv)

The Vision Transformer (ViT) demonstrates exceptional performance in various computer vision tasks. Attention is crucial for ViT to capture complex wide-ranging relationships among image patches, allowing the model to weigh the importance of image patches and aiding our understanding of the decision-making process. However, when utilizing the attention of ViT as evidence in high-stakes decision-making tasks such as medical diagnostics, a challenge arises due to the potential of attention mechanisms erroneously focusing on irrelevant regions. In this study, we propose a statistical test for ViT's attentions, enabling us to use the attentions as reliable quantitative evidence indicators for ViT's decision-making with a rigorously controlled error rate. Using the framework called selective inference, we quantify the statistical significance of attentions in the form of p-values, which enables the theoretically grounded quantification of the false positive detection probability of attentions. We demonstrate the validity and the effectiveness of the proposed method through numerical experiments and applications to brain image diagnoses.