Sharper Generalization Bounds for Transformer
Not on the map. Why?
Abstract (arXiv)
This paper studies generalization error bounds for Transformer models. Based on the offset Rademacher complexity, we derive sharper generalization bounds for different Transformer architectures, including single-layer single-head, single-layer multi-head, and multi-layer Transformers. We first express the excess risk of Transformers in terms of the offset Rademacher complexity. By exploiting its connection with the empirical covering numbers of the corresponding hypothesis spaces, we obtain excess risk bounds that achieve optimal convergence rates up to constant factors. We then derive refined excess risk bounds by upper bounding the covering numbers of Transformer hypothesis spaces using matrix ranks and matrix norms, leading to precise, architecture-dependent generalization bounds. Finally, we relax the boundedness assumption on feature mappings and extend our theoretical results to settings with unbounded (sub-Gaussian) features and heavy-tailed distributions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 23 Mar 2026.
Builds on
29 of 40 references.
References · 29
- 2017Attention Is All You Needcited by 59,932
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2023A Survey on Evaluation of Large Language Modelscited by 1,227
- 2020A Survey on Visual Transformercited by 653
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
- 2015An Introduction to Matrix Concentration Inequalitiescited by 391
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2021User-friendly introduction to PAC-Bayes boundscited by 234
- 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
- 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
- 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
- 2024A Formal Framework for Understanding Length Generalization in Transformerscited by 44
17 more
- 2015Learning with Square Loss: Localization through Offset Rademacher Complexitycited by 37
- 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
- 2018Finite-sample analysis of M-estimators using self-concordancecited by 30
- 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
- 2023Sequence Length Independent Norm-Based Generalization Bounds for Transformerscited by 28
- 2021Convergence rates of deep ReLU networks for multiclass classificationcited by 21
- 2024On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networkscited by 13
- 2024Length-Induced Embedding Collapse in PLM-based Modelscited by 9
- 2022Non-Asymptotic Guarantees for Robust Statistical Learning under Infinite Variance Assumptioncited by 6
- 2024Semi-Supervised Deep Sobolev Regression: Estimation and Variable Selection by ReQU Neural Networkcited by 5
- 2024On Generalization Bounds for Neural Networks with Low Rank Layerscited by 4
- 2025Uniform Mean Estimation for Heavy-Tailed Distributions via Median-of-Meanscited by 4
- 2024On Generalisation Error Bounds for Transformerscited by 3
- 2025Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfittingcited by 2
- 2025Generalization Bounds for Rank-sparse Neural Networkscited by 2
- 2025From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputscited by 1
- 2026Generalization Bounds for Transformer Channel Decoderscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.