# Sharper Generalization Bounds for Transformer

arXiv: https://arxiv.org/abs/2603.21541 · HTML: https://mlsanity.com/paper/2603.21541/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Chaining Mutual Information and Tightening Generalization Bounds](https://mlsanity.com/paper/1806.03803/) (2018)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [A Formal Framework for Understanding Length Generalization in Transformers](https://mlsanity.com/paper/2410.02140/) (2024)
- [Learning with Square Loss: Localization through Offset Rademacher Complexity](https://mlsanity.com/paper/1502.06134/) (2015)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Finite-sample analysis of M-estimators using self-concordance](https://mlsanity.com/paper/1810.06838/) (2018)
- [Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data](https://mlsanity.com/paper/2411.06646/) (2024)
- [Sequence Length Independent Norm-Based Generalization Bounds for Transformers](https://mlsanity.com/paper/2310.13088/) (2023)
- [Convergence rates of deep ReLU networks for multiclass classification](https://mlsanity.com/paper/2108.00969/) (2021)
- [On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks](https://mlsanity.com/paper/2409.00901/) (2024)
- [Length-Induced Embedding Collapse in PLM-based Models](https://mlsanity.com/paper/2410.24200/) (2024)
- [Non-Asymptotic Guarantees for Robust Statistical Learning under Infinite Variance Assumption](https://mlsanity.com/paper/2201.03182/) (2022)
- [Semi-Supervised Deep Sobolev Regression: Estimation and Variable Selection by ReQU Neural Network](https://mlsanity.com/paper/2401.04535/) (2024)
- [On Generalisation Error Bounds for Transformers](https://mlsanity.com/paper/2410.11500/) (2024)
- [On Generalization Bounds for Neural Networks with Low Rank Layers](https://mlsanity.com/paper/2411.13733/) (2024)
- [Uniform Mean Estimation for Heavy-Tailed Distributions via Median-of-Means](https://mlsanity.com/paper/2506.14673/) (2025)
- [Generalization Bounds for Rank-sparse Neural Networks](https://mlsanity.com/paper/2510.21945/) (2025)
- [Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfitting](https://mlsanity.com/paper/2502.12508/) (2025)
- [Generalization Bounds for Transformer Channel Decoders](https://mlsanity.com/paper/2601.06969/) (2026)
- [From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputs](https://mlsanity.com/paper/2512.12805/) (2025)

## Built on by (most cited first, 3 of 3)

- [Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers](https://mlsanity.com/paper/2605.07297/) (2026)
- [Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustness](https://mlsanity.com/paper/2607.27975/) (2026)
- [VC Dimension and Expressivity of Real-Valued Transformers](https://mlsanity.com/paper/2609.31999/) (2026)

## Abstract (arXiv)

This paper studies generalization error bounds for Transformer models. Based on the offset Rademacher complexity, we derive sharper generalization bounds for different Transformer architectures, including single-layer single-head, single-layer multi-head, and multi-layer Transformers. We first express the excess risk of Transformers in terms of the offset Rademacher complexity. By exploiting its connection with the empirical covering numbers of the corresponding hypothesis spaces, we obtain excess risk bounds that achieve optimal convergence rates up to constant factors. We then derive refined excess risk bounds by upper bounding the covering numbers of Transformer hypothesis spaces using matrix ranks and matrix norms, leading to precise, architecture-dependent generalization bounds. Finally, we relax the boundedness assumption on feature mappings and extend our theoretical results to settings with unbounded (sub-Gaussian) features and heavy-tailed distributions.