# One-layer transformers fail to solve the induction heads task

arXiv: https://arxiv.org/abs/2408.14332 · HTML: https://mlsanity.com/paper/2408.14332/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Birth of a Transformer: A Memory Viewpoint](https://mlsanity.com/paper/2306.00802/) (2023)
- [Representational Strengths and Limitations of Transformers](https://mlsanity.com/paper/2306.02896/) (2023)
- [Transformers, parallel computation, and logarithmic depth](https://mlsanity.com/paper/2402.09268/) (2024)
- [On Limitations of the Transformer Architecture](https://mlsanity.com/paper/2402.08164/) (2024)

## Built on by (most cited first, 30 of 32)

- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [Theoretical limitations of multi-layer Transformer](https://mlsanity.com/paper/2412.02975/) (2024)
- [How Do LLMs Perform Two-Hop Reasoning in Context?](https://mlsanity.com/paper/2502.13913/) (2025)
- [Toward Understanding In-context vs. In-weight Learning](https://mlsanity.com/paper/2410.23042/) (2024)
- [From Markov to Laplace: How Mamba In-Context Learns Markov Chains](https://mlsanity.com/paper/2502.10178/) (2025)
- [Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers](https://mlsanity.com/paper/2506.10887/) (2025)
- [Lost in Transmission: When and Why LLMs Fail to Reason Globally](https://mlsanity.com/paper/2505.08140/) (2025)
- [Sequences of Logits Reveal the Low Rank Structure of Language Models](https://mlsanity.com/paper/2510.24966/) (2025)
- [How Transformers Get Rich: Approximation and Dynamics Analysis](https://mlsanity.com/paper/2410.11474/) (2024)
- [KV Shifting Attention Enhances Language Modeling](https://mlsanity.com/paper/2411.19574/) (2024)
- [Revisiting associative recall in modern recurrent models](https://mlsanity.com/paper/2508.19029/) (2025)
- [Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity](https://mlsanity.com/paper/2506.11891/) (2025)
- [Predicting the Emergence of Induction Heads in Language Model Pretraining](https://mlsanity.com/paper/2511.16893/) (2025)
- [The Transformer Cookbook](https://mlsanity.com/paper/2510.00368/) (2025)
- [Decoupling Positional and Symbolic Attention Behavior in Transformers](https://mlsanity.com/paper/2511.11579/) (2025)
- [From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers](https://mlsanity.com/paper/2512.18634/) (2025)
- [ENTP: Encoder-only Next Token Prediction](https://mlsanity.com/paper/2410.01600/) (2024)
- [On the Emergence of Induction Heads for In-Context Learning](https://mlsanity.com/paper/2511.01033/) (2025)
- [Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)](https://mlsanity.com/paper/2503.22076/) (2025)
- [Fast attention mechanisms: a tale of parallelism](https://mlsanity.com/paper/2509.09001/) (2025)
- [Energy-Entropy Regularization: The True Power of Minimal Looped Transformers](https://mlsanity.com/paper/2601.09588/) (2026)
- [Indexing: the Beginning and the End](https://mlsanity.com/paper/2607.22361/) (2026)
- [A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention](https://mlsanity.com/paper/2602.01763/) (2026)
- [Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers](https://mlsanity.com/paper/2602.02707/) (2026)
- [Attention-based representations for multi-task computation](https://mlsanity.com/paper/2608.04243/) (2026)
- [Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective](https://mlsanity.com/paper/2510.03784/) (2025)
- [Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Induction](https://mlsanity.com/paper/2602.07562/) (2026)
- [On the Computational Hardness of Transformers](https://mlsanity.com/paper/2603.11332/) (2026)
- [InfoFlow: A Framework for Multi-Layer Transformer Analysis](https://mlsanity.com/paper/2605.17930/) (2026)
- [Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction](https://mlsanity.com/paper/2609.32319/) (2026)

## Abstract (arXiv)

A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.