mlsanityData to 7 Oct 2026

Understanding LLM Failures: A Multi-Tape Turing Machine Analysis of Systematic Errors in Language Model Reasoning

arXiv 2602.15868 ↗First posted 27 Jan 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) exhibit failure modes on seemingly trivial tasks. We propose a formalisation of LLM interaction using a deterministic multi-tape Turing machine, where each tape represents a distinct component: input characters, tokens, vocabulary, model parameters, activations, probability distributions, and output text. The model enables precise localisation of failure modes to specific pipeline stages, revealing, e.g., how tokenisation obscures character-level structure needed for counting tasks. The model clarifies why techniques like chain-of-thought prompting help, by externalising computation on the output tape, while also revealing their fundamental limitations. This approach provides a rigorous, falsifiable alternative to geometric metaphors and complements empirical scaling laws with principled error analysis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 21 of its 31 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

21 of 31 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2014Neural Turing Machinescited by 630
  8. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  9. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  10. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  11. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  12. 2022Transformers Learn Shortcuts to Automatacited by 260
9 more
  1. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  2. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  3. 2021Thinking Like Transformerscited by 182
  4. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  5. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  6. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 100
  7. 2024Why Do Large Language Models (LLMs) Struggle to Count Letters?cited by 37
  8. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilitiescited by 18
  9. 2024Autoregressive Large Language Models are Computationally Universalcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Large Language Model Reasoning Failures
  2. 2026Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning
  3. 2026Beyond Memorization: Testing LLM Reasoning on Unseen Theory of Computation Tasks
  4. 2026From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks
  5. 2026How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures
  6. 2026Dissecting Failure Dynamics in Large Language Model Reasoning
  7. 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
  8. 2024LLM2: Let Large Language Models Harness System 2 Reasoning