# Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors

arXiv: https://arxiv.org/abs/2603.14087 · HTML: https://mlsanity.com/paper/2603.14087/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 14 Mar 2026.

## Builds on (most cited first, 30 of 34)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [LLM Circuit Analyses Are Consistent Across Training and Scale](https://mlsanity.com/paper/2407.10827/) (2024)
- [Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models](https://mlsanity.com/paper/2403.15498/) (2024)
- [Do language models plan ahead for future tokens?](https://mlsanity.com/paper/2404.00859/) (2024)
- [Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient](https://mlsanity.com/paper/2410.02984/) (2024)
- [Evidence of Learned Look-Ahead in a Chess-Playing Neural Network](https://mlsanity.com/paper/2406.00877/) (2024)
- [Emergent Representations of Program Semantics in Language Models Trained on Programs](https://mlsanity.com/paper/2305.11169/) (2023)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [Tracking the Feature Dynamics in LLM Training: A Mechanistic Study](https://mlsanity.com/paper/2412.17626/) (2024)
- [Hidden Breakthroughs in Language Model Training](https://mlsanity.com/paper/2506.15872/) (2025)
- [The Belief State Transformer](https://mlsanity.com/paper/2410.23506/) (2024)

## Abstract (arXiv)

Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model. Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training.