mlsanityData to 7 Oct 2026

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

arXiv 2402.00522 ↗First posted 1 Feb 2024cs.LG stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads. These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 24 of its 79 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

24 of 79 references.

References · 24

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  10. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  11. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  12. 2022Transformers Learn Shortcuts to Automatacited by 260
12 more
  1. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  2. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  3. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  4. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  5. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  6. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  7. 2023Transformers learn through gradual rank increasecited by 52
  8. 2023The Expressive Power of Transformers with Chain of Thoughtcited by 47
  9. 2023Approximation Rate of the Transformer Architecture for Sequence Modelingcited by 28
  10. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  11. 2024Anchor function: a type of benchmark functions for studying language modelscited by 10
  12. 2022Why self-attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetriescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 28
  2. 2024Measure-to-measure interpolation using Transformerscited by 28
  3. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  4. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  5. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
  6. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  7. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  8. 2024Towards Understanding the Universality of Transformers for Next-Token Predictioncited by 9
  9. 2025An explainable transformer circuit for compositional generalizationcited by 8
  10. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7
  11. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  12. 2025Transformers through the lens of support-preserving maps between measurescited by 5
13 more
  1. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4
  2. 2025Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Modelscited by 4
  3. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  4. 2025The Effect of Attention Head Count on Transformer Approximationcited by 4
  5. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  6. 2026More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activationscited by 3
  7. 2025A unified framework for establishing the universal approximation of transformer-type architecturescited by 2
  8. 2026Affine-Scaled Attention: Towards Flexible and Stable Transformer Attentioncited by 2
  9. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1
  10. 2026Geometric Capacity of Transformers: A Tropical Geometry Perspectivecited by 1
  11. 2026Measure-to-measure Regression with Transformerscited by 1
  12. 2025Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspectivecited by 0
  13. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0