mlsanityData to 14 Oct 2026

Your Context Is Not an Array: Unveiling Random Access Limitations in Transformers

arXiv 2408.05506 ↗First posted 10 Aug 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Despite their recent successes, Transformer-based large language models show surprising failure modes. A well-known example of such failure modes is their inability to length-generalize: solving problem instances at inference time that are longer than those seen during training. In this work, we further explore the root cause of this failure by performing a detailed analysis of model behaviors on the simple parity task. Our analysis suggests that length generalization failures are intricately related to a model's inability to perform random memory accesses within its context window. We present supporting evidence for this hypothesis by demonstrating the effectiveness of methodologies that circumvent the need for indexing or that enable random token access indirectly, through content-based addressing. We further show where and how the failure to perform random memory access manifests through attention map visualizations.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 34 of its 36 references that are in the map. First posted on arXiv 10 Aug 2024.

Builds on

34 of 36 references.

References · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
22 more
  1. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  3. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  4. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  5. 2022Autoformalization with Large Language Modelscited by 263
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  7. 2022Exploring Length Generalization in Large Language Modelscited by 206
  8. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  9. 2021Thinking Like Transformerscited by 184
  10. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  11. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
  12. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  13. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  14. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  15. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  16. 2023Natural Language Reasoning, A Surveycited by 61
  17. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  18. 2020The EOS Decision and Length Extrapolationcited by 46
  19. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  20. 2021Teaching Autoregressive Language Models Complex Tasks By Demonstrationcited by 15
  21. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  22. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 46
  2. 2025Lost in Transmission: When and Why LLMs Fail to Reason Globallycited by 9
  3. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  4. 2025Revisiting Bi-Linear State Transitions in Recurrent Neural Networkscited by 4
  5. 2024Compositional Hardness of Code in Large Language Models -- A Probabilistic Perspectivecited by 3
  6. 2025Delayed Attention Training Improves Length Generalization in Transformer--RNN Hybridscited by 2
  7. 2026LUMINA: Long-horizon Understanding for Multi-turn Interactive Agentscited by 2
  8. 2026On the "Induction Bias" in Sequence Modelscited by 2
  9. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1
  10. 2026Barriers to Universal Reasoning With Transformers (And How to Overcome Them)cited by 1
  11. 2025A Group Theoretic Analysis of the Symmetries Underlying Base Addition and Their Learnability by Neural Networkscited by 0