mlsanityData to 7 Oct 2026

One-layer transformers fail to solve the induction heads task

arXiv 2408.14332 ↗First posted 26 Aug 2024cs.LG stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 7 of its 11 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

7 of 11 references.

References · 7

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  5. 2023Representational Strengths and Limitations of Transformerscited by 132
  6. 2024Transformers, parallel computation, and logarithmic depthcited by 75
  7. 2024On Limitations of the Transformer Architecturecited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 16; 2026: 9.

30 papers.

Sort:

By area:

  1. 2025On the Emergence of Position Bias in Transformerscited by 57
  2. 2024Theoretical limitations of multi-layer Transformercited by 43
  3. 2025How Do LLMs Perform Two-Hop Reasoning in Context?cited by 20
  4. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  5. 2025Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformerscited by 10
  6. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 9
  7. 2025Lost in Transmission: When and Why LLMs Fail to Reason Globallycited by 9
  8. 2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 8
  9. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7
  10. 2024KV Shifting Attention Enhances Language Modelingcited by 7
  11. 2025Revisiting associative recall in modern recurrent modelscited by 7
  12. 2025Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacitycited by 6
18 more
  1. 2025Decoupling Positional and Symbolic Attention Behavior in Transformerscited by 6
  2. 2025From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformerscited by 6
  3. 2024ENTP: Encoder-only Next Token Predictioncited by 5
  4. 2025The Transformer Cookbookcited by 5
  5. 2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 5
  6. 2025On the Emergence of Induction Heads for In-Context Learningcited by 3
  7. 2025Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)cited by 2
  8. 2025Fast attention mechanisms: a tale of parallelismcited by 2
  9. 2026Energy-Entropy Regularization: The True Power of Minimal Looped Transformerscited by 2
  10. 2026A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attentioncited by 1
  11. 2026Indexing: the Beginning and the Endcited by 1
  12. 2026Attention-based representations for multi-task computationcited by 1
  13. 2025Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspectivecited by 0
  14. 2026Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformerscited by 0
  15. 2026Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Inductioncited by 0
  16. 2026On the Computational Hardness of Transformerscited by 0
  17. 2026InfoFlow: A Framework for Multi-Layer Transformer Analysiscited by 0
  18. 2026Why Do Conventional World Models Fail to Learn Cellular Automata?cited by 0