What do language models model? Transformers, automata, and the format of thought
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
What do large language models actually model? Do they tell us something about human capacities, or are they models of the corpus we've trained them on? I give a non-deflationary defence of the latter position. Cognitive science tells us that linguistic capabilities in humans rely supralinear formats for computation. The transformer architecture, by contrast, supports at best a linear formats for processing. This argument will rely primarily on certain invariants of the computational architecture of transformers. I then suggest a positive story about what transformers are doing, focusing on Liu et al. (2022)'s intriguing speculations about shortcut automata. I conclude with why I don't think this is a terribly deflationary story. Language is not (just) a means for expressing inner state but also a kind of 'discourse machine' that lets us make new language given appropriate context. We have learned to use this technology in one way; LLMs have also learned to use it too, but via very different means.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 56 references that are in the map. First posted on arXiv 26 Aug 2025.
Builds on
12 of 56 references.
References · 12
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
- 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
- 2022Transformers Learn Shortcuts to Automatacited by 268
- 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 104
- 2021Position Information in Transformers: An Overviewcited by 103
- 2023Could a Large Language Model be Conscious?cited by 73
- 2023From task structures to world models: What do LLMs know?cited by 46
- 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debatescited by 23
- 2023Permutation Equivariance of Transformers and Its Applicationscited by 20
- 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Understanding Large Language Models
- 2025Three tiers of computation in transformers and in brain architectures
- 2025Evaluating Large Language Models on the Frame and Symbol Grounding Problems: A Zero-shot Benchmark
- 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition
- 2024Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer
- 2026LLM and Human Modes of Representation
- 2026LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
- 2022What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured