# Benign Overfitting in Token Selection of Attention Mechanism

arXiv: https://arxiv.org/abs/2409.17625 · HTML: https://mlsanity.com/paper/2409.17625/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2024.

## Built on by (most cited first, 3 of 3)

- [Benign Overfitting in Adversarial Training for Vision Transformers](https://mlsanity.com/paper/2604.19724/) (2026)
- [When Does ℓ2-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ℓ1 Implicit Bias](https://mlsanity.com/paper/2605.06314/) (2026)
- [Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models](https://mlsanity.com/paper/2603.22801/) (2026)

## Abstract (arXiv)

Attention mechanism is a fundamental component of the transformer model and plays a significant role in its success. However, the theoretical understanding of how attention learns to select tokens is still an emerging area of research. In this work, we study the training dynamics and generalization ability of the attention mechanism under classification problems with label noise. We show that, with the characterization of signal-to-noise ratio (SNR), the token selection of attention mechanism achieves benign overfitting, i.e., maintaining high generalization performance despite fitting label noise. Our work also demonstrates an interesting delayed acquisition of generalization after an initial phase of overfitting. Finally, we provide experiments to support our theoretical analysis using both synthetic and real-world datasets.