# Language models are better than humans at next-token prediction

arXiv: https://arxiv.org/abs/2212.11281 · HTML: https://mlsanity.com/paper/2212.11281/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 17 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 4 of 4)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)

## Built on by (most cited first, 15 of 15)

- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [Extending Activation Steering to Broad Skills and Multiple Behaviours](https://mlsanity.com/paper/2403.05767/) (2024)
- [From Language to Cognition: How LLMs Outgrow the Human Language Network](https://mlsanity.com/paper/2503.01830/) (2025)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System](https://mlsanity.com/paper/2509.10540/) (2025)
- [Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains](https://mlsanity.com/paper/2311.07723/) (2023)
- [Multimodal Contrastive In-Context Learning](https://mlsanity.com/paper/2408.12959/) (2024)
- [From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation](https://mlsanity.com/paper/2408.02769/) (2024)
- [Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attention](https://mlsanity.com/paper/2501.06382/) (2025)
- [Multiple Choice Learning of Low-Rank Adapters for Language Modeling](https://mlsanity.com/paper/2507.10419/) (2025)
- [Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis](https://mlsanity.com/paper/2506.22516/) (2025)
- [Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal](https://mlsanity.com/paper/2601.09886/) (2026)
- [Post-Norm can Resharpen Attention](https://mlsanity.com/paper/2510.08341/) (2025)
- [Provable Long-Range Benefits of Next-Token Prediction](https://mlsanity.com/paper/2512.07818/) (2025)
- [AI Models Can Predict and Collaboratively Modulate Human Memory Search](https://mlsanity.com/paper/2608.26152/) (2026)

## Abstract (arXiv)

Current language models are considered to have sub-human capabilities at natural language tasks like question-answering or writing code. However, language models are not trained to perform well at these tasks, they are trained to accurately predict the next token given previous tokes in tokenized text. It is not clear whether language models are better or worse than humans at next token prediction. To try to answer this question, we performed two distinct experiments to directly compare humans and language models on this front: one measuring top-1 accuracy and the other measuring perplexity. In both experiments, we find humans to be consistently worse than even relatively small language models like GPT3-Ada at next-token prediction.