mlsanityData to 14 Oct 2026

Sequence Length Independent Norm-Based Generalization Bounds for Transformers

arXiv 2310.13088 ↗First posted 19 Oct 2023stat.ML cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

This paper provides norm-based generalization bounds for the Transformer architecture that do not depend on the input sequence length. We employ a covering number based approach to prove our bounds. We use three novel covering number bounds for the function class of bounded linear transformations to upper bound the Rademacher complexity of the Transformer. Furthermore, we show this generalization bound applies to the common Transformer training technique of masking and then predicting the masked word. We also run a simulated study on a sparse majority data set that empirically validates our theoretical findings.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 16 of its 42 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

16 of 42 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2016Layer Normalizationcited by 5,669
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  6. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  7. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  8. 2020Generalization and Representational Limits of Graph Neural Networkscited by 218
  9. 2019Understanding and Improving Layer Normalizationcited by 217
  10. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  11. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  12. 2022Mitigating Memorization of Noisy Labels by Clipping the Model Predictioncited by 40
4 more
  1. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 40
  2. 2019Generalization Bounds for Convolutional Neural Networkscited by 21
  3. 2022On Rademacher Complexity-based Generalization Bounds for Deep Learningcited by 16
  4. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 11.

29 papers.

Sort:

By area:

  1. 2024Provably learning a multi-head attention layercited by 21
  2. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  3. 2025LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selectioncited by 14
  4. 2024On the Optimal Memorization Capacity of Transformerscited by 10
  5. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  6. 2024Learning Linear Attention in Polynomial Timecited by 5
  7. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attentioncited by 5
  8. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  9. 2024On Generalisation Error Bounds for Transformerscited by 4
  10. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 3
  11. 2026Sharper Generalization Bounds for Transformercited by 3
  12. 2026Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexitycited by 3
17 more
  1. 2024Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraintscited by 2
  2. 2025Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regressioncited by 2
  3. 2026Generalization Bounds for Transformer Channel Decoderscited by 2
  4. 2026Tight Sample Complexity of Transformerscited by 2
  5. 2025A Theory for Conditional Generative Modeling on Multiple Data Sourcescited by 1
  6. 2025From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputscited by 1
  7. 2026SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Modelscited by 1
  8. 2026Lossless Anti-Distillation Samplingcited by 1
  9. 2025Generalizability of Neural Networks Minimizing Empirical Risk Based on Expressive Abilitycited by 0
  10. 2025Forecasting Geopolitical Events with a Sparse Temporal Fusion Transformer and Gaussian Process Hybrid: A Case Study in Middle Eastern and U.S. Conflict Dynamicscited by 0
  11. 2025Effective Sample Size and Generalization Bounds for Temporal Networkscited by 0
  12. 2025Dynamical Implicit Neural Representationscited by 0
  13. 2026ARMOR: Adaptive Retriever Optimization for Low-Resource Telecom Question Answeringcited by 0
  14. 2026Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimationcited by 0
  15. 2026Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustnesscited by 0
  16. 2026VC Dimension and Expressivity of Real-Valued Transformerscited by 0
  17. 2026On the Trade-off Between Information Loss and Generalization in Sparse Attentioncited by 0