mlsanityData to 7 Oct 2026

Exposing Attention Glitches with Flip-Flop Language Modeling

arXiv 2306.00946 ↗First posted 1 Jun 2023cs.LG cs.CLcited by 74

Not on the map. Why?

Abstract (arXiv)

Why do large language models sometimes output factual inaccuracies and exhibit erroneous reasoning? The brittleness of these models, particularly when executing long chains of reasoning, currently seems to be an inevitable price to pay for their advanced capabilities of coherently synthesizing knowledge, pragmatics, and abstract thought. Towards making sense of this fundamentally unsolved problem, this work identifies and analyzes the phenomenon of attention glitches, in which the Transformer architecture's inductive biases intermittently fail to capture robust reasoning. To isolate the issue, we introduce flip-flop language modeling (FFLM), a parametric family of synthetic benchmarks designed to probe the extrapolative behavior of neural language models. This simple generative task requires a model to copy binary symbols over long-range dependencies, ignoring the tokens in between. We find that Transformer FFLMs suffer from a long tail of sporadic reasoning errors, some of which we can eliminate using various regularization techniques. Our preliminary mechanistic analyses show why the remaining errors may be very difficult to diagnose and resolve. We hypothesize that attention glitches account for (some of) the closed-domain hallucinations in natural LLMs.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 77 of its 109 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

77 of 109 references.

References · 77

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
38 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  8. 2019Attention is not Explanationcited by 916
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  13. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  14. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  15. 2014Neural Turing Machinescited by 630
  16. 2018Universal Transformerscited by 613
  17. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  18. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  19. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  20. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  21. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  22. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  23. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  24. 2023How Language Model Hallucinations Can Snowballcited by 310
  25. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  26. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  27. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  28. 2022Transformers Learn Shortcuts to Automatacited by 260
  29. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  30. 2022Memorizing Transformerscited by 219
  31. 2022Exploring Length Generalization in Large Language Modelscited by 204
  32. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  33. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  34. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  35. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  36. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  37. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  38. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 26; 2025: 27; 2026: 12.

74 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  5. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  6. 2023Teaching Arithmetic to Small Transformerscited by 121
  7. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  8. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  9. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  10. 2024Can large language models explore in-context?cited by 75
  11. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
  12. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
38 more
  1. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  2. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  3. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  4. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  5. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 31
  6. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  7. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  8. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  9. 2025Long-Context Generalization with Sparse Attentioncited by 18
  10. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  11. 2024Simulating Hard Attention Using Soft Attentioncited by 15
  12. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  13. 2024Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursioncited by 14
  14. 2023In-Context Learning Dynamics with Random Binary Sequencescited by 13
  15. 2025Knee-Deep in C-RASP: A Transformer Depth Hierarchycited by 12
  16. 2024Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Taskscited by 10
  17. 2025Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillationcited by 10
  18. 2025On the Query Complexity of Verifier-Assisted Language Generationcited by 10
  19. 2025Unique Hard Attention: A Tale of Two Sidescited by 10
  20. 2025Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decodingcited by 10
  21. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  22. 2023An Investigation of LLMs' Inefficacy in Understanding Converse Relationscited by 9
  23. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  24. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  25. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  26. 2024Addressing Image Hallucination in Text-to-Image Generation through Factual Image Retrievalcited by 6
  27. 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilitiescited by 6
  28. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  29. 2023Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalabilitycited by 5
  30. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  31. 2024Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learningcited by 4
  32. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  33. 2024Transformer Mechanisms Mimic Frontostriatal Gating Operations When Trained on Human Working Memory Taskscited by 3
  34. 2024CogMG: Collaborative Augmentation Between Large Language Model and Knowledge Graphcited by 3
  35. 2025Recency Biased Causal Attention for Time-series Forecastingcited by 3
  36. 2025LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layerscited by 3
  37. 2026On the Spatiotemporal Dynamics of Generalization in Neural Networkscited by 3
  38. 2025Attend or Perish: Benchmarking Attention in Algorithmic Reasoningcited by 2

50 most cited of 74 All 74 (JSON)