mlsanityData to 14 Oct 2026

A model of errors in transformers

arXiv 2601.14175 ↗First posted 20 Jan 2026cs.LG cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

We study the error rate of LLMs on tasks like arithmetic that require a deterministic output, and repetitive processing of tokens drawn from a small set of alternatives. We argue that incorrect predictions arise when small errors in the attention mechanism accumulate to cross a threshold, and use this insight to derive a quantitative two-parameter relationship between the accuracy and the complexity of the task. The two parameters vary with the prompt and the model; they can be interpreted in terms of an elementary noise rate, and the number of plausible erroneous tokens that can be predicted. Our analysis is inspired by an ``effective field theory'' perspective: the LLM's many raw parameters can be reorganized into just two parameters that govern the error rate. We perform extensive empirical tests, using Gemini 2.5 Flash, Gemini 2.5 Pro and DeepSeek R1, and find excellent agreement between the predicted and observed accuracy for a variety of tasks, although we also identify deviations in some cases. Our model provides an alternative to suggestions that errors made by LLMs on long repetitive tasks indicate the ``collapse of reasoning'', or an inability to express ``compositional'' functions. Finally, we show how to construct prompts to reduce the error rate.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 20 Jan 2026.

Builds on

44 of 58 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2016Layer Normalizationcited by 5,669
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  12. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
32 more
  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  5. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  7. 2024A Survey on Large Language Models for Code Generationcited by 658
  8. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  9. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  10. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  11. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  12. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  13. 2021Thinking Like Transformerscited by 184
  14. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  15. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  16. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  17. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  18. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 97
  19. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  20. 2024On Limitations of the Transformer Architecturecited by 63
  21. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  22. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 53
  23. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 45
  24. 2024Theoretical limitations of multi-layer Transformercited by 45
  25. 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
  26. 2024How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMscited by 24
  27. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  28. 2025Constant Bit-size Transformers Are Turing Completecited by 19
  29. 2025Probing for Arithmetic Errors in Language Modelscited by 18
  30. 2023Arithmetic with Language Models: from Memorization to Computationcited by 12
  31. 2025The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate Itcited by 10
  32. 2025Softmax Transformers are Turing-Completecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026XLGoBench: Detecting cross-lingual skill gaps with algorithmic taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
  2. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Tasks
  3. 2026Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning
  4. 2024LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems
  5. 2025Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic
  6. 2026Represented Is Not Computed: A Causal Test of Candidate Algorithmic Intermediates in a Transformer
  7. 2026Understanding LLM Failures: A Multi-Tape Turing Machine Analysis of Systematic Errors in Language Model Reasoning
  8. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Models