mlsanityData to 14 Oct 2026

Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective

arXiv 2510.03784 ↗First posted 4 Oct 2025cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformers have achieved remarkable successes across a wide range of applications, yet the theoretical foundation of their model efficiency remains underexplored. In this work, we investigate how the model parameters -- mainly attention heads and head dimensions -- should be allocated across layers to balance expressivity and efficiency. We first provide mathematical analysis on the role of early layers in information extraction from an approximation perspective, with a theoretical characterization on the trade-off between the number of heads and head dimension under a fixed parameter budget. In addition, we uncover and prove the saturation behavior of softmax activations: Continuously increasing head dimensions can lead to diminishing returns in learning errors, particularly for long sequences. Supported by both theory and experiments, this saturation pattern suggests that later layers can operate more efficiently with reduced parameters. Combining these insights, we propose principled strategies for allocating attention heads and dimensions across Transformers' layers, shedding light on theoretically-grounded model efficiency of Transformer-based architectures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 70 of its 96 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

70 of 96 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2020Longformer: The Long-Document Transformercited by 3,413
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  5. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  6. 2020Rethinking Attention with Performerscited by 1,601
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2018Self-Attention with Relative Position Representationscited by 1,120
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2022In-context Learning and Induction Headscited by 964
  12. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  13. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  14. 2019Are Sixteen Heads Really Better than One?cited by 816
  15. 2022Transformers learn in-context by gradient descentcited by 732
  16. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  17. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  18. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  19. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  20. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  21. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  22. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  23. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  24. 2022Neighborhood Attention Transformercited by 239
  25. 2023Looped Transformers as Programmable Computerscited by 237
  26. 2021Thinking Like Transformerscited by 184
  27. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  28. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  29. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  30. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
  31. 2023Large Language Models for Robotics: A Surveycited by 133
  32. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 95
  33. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
  34. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 79
  35. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  36. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  37. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  38. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 47

50 most cited of 70 All 70 (JSON)

Built on by

Not cited yet.