# Sequential Enumeration in Large Language Models

arXiv: https://arxiv.org/abs/2512.04727 · HTML: https://mlsanity.com/paper/2512.04727/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Builds on (most cited first, 25 of 25)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Chain of Thought Empowers Transformers to Solve Inherently Serial Problems](https://mlsanity.com/paper/2402.12875/) (2024)
- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [Exploring Length Generalization in Large Language Models](https://mlsanity.com/paper/2207.04901/) (2022)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [On the Practical Computational Power of Finite Precision RNNs for Language Recognition](https://mlsanity.com/paper/1805.04908/) (2018)
- [Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem](https://mlsanity.com/paper/2411.00238/) (2024)
- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [Why Do Large Language Models (LLMs) Struggle to Count Letters?](https://mlsanity.com/paper/2412.18626/) (2024)
- [Counting Ability of Large Language Models and Impact of Tokenization](https://mlsanity.com/paper/2410.19730/) (2024)
- [Language Models Need Inductive Biases to Count Inductively](https://mlsanity.com/paper/2405.20131/) (2024)
- [When Can Transformers Count to n?](https://mlsanity.com/paper/2407.15160/) (2024)
- [Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models](https://mlsanity.com/paper/2303.07735/) (2023)
- [Counting and Algorithmic Generalization with Transformers](https://mlsanity.com/paper/2310.08661/) (2023)
- [The Unreasonable Effectiveness of Eccentric Automatic Prompts](https://mlsanity.com/paper/2402.10949/) (2024)
- [Evaluating Numerical Reasoning in Text-to-Image Models](https://mlsanity.com/paper/2406.14774/) (2024)
- [Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies](https://mlsanity.com/paper/2402.17396/) (2024)
- [Visual Enumeration Remains Challenging for Multimodal Generative AI](https://mlsanity.com/paper/2402.03328/) (2024)
- [Modelling the development of counting with memory-augmented neural networks](https://mlsanity.com/paper/2105.10577/) (2021)
- [Zero-shot counting with a dual-stream neural network model](https://mlsanity.com/paper/2405.09953/) (2024)

## Built on by (most cited first, 3 of 3)

- [Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets](https://mlsanity.com/paper/2608.01000/) (2026)
- [Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models](https://mlsanity.com/paper/2512.15254/) (2025)
- [The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale](https://mlsanity.com/paper/2609.26087/) (2026)

## Abstract (arXiv)

Reliably counting and generating sequences of items remain a significant challenge for neural networks, including Large Language Models (LLMs). Indeed, although this capability is readily handled by rule-based symbolic systems based on serial computation, learning to systematically deploy counting procedures is difficult for neural models, which should acquire these skills through learning. Previous research has demonstrated that recurrent architectures can only approximately track and enumerate sequences of events, and it remains unclear whether modern deep learning systems, including LLMs, can deploy systematic counting procedures over sequences of discrete symbols. This paper aims to fill this gap by investigating the sequential enumeration abilities of five state-of-the-art LLMs, including proprietary, open-source, and reasoning models. We probe LLMs in sequential naming and production tasks involving lists of letters and words, adopting a variety of prompting instructions to explore the role of chain-of-thought in the spontaneous emerging of counting strategies. We also evaluate open-source models with the same architecture but increasing size to see whether the mastering of counting principles follows scaling laws, and we analyze the embedding dynamics during sequential enumeration to investigate the emergent encoding of numerosity. We find that some LLMs are indeed capable of deploying counting procedures when explicitly prompted to do so, but none of them spontaneously engage in counting when simply asked to enumerate the number of items in a sequence. Our results suggest that, despite their impressive emergent abilities, LLMs cannot yet robustly and systematically deploy counting procedures, highlighting a persistent gap between neural and symbolic approaches to compositional generalization.