mlsanityData to 14 Oct 2026

Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors

arXiv 2603.14087 ↗First posted 14 Mar 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model. Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 14 Mar 2026.

Builds on

34 of 46 references.

References · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  5. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  9. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  11. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  12. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
22 more
  1. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  2. 2023Language Models Represent Space and Timecited by 316
  3. 2022Transformers Learn Shortcuts to Automatacited by 268
  4. 2024The pitfalls of next-token predictioncited by 160
  5. 2023The Quantization Model of Neural Scalingcited by 138
  6. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  7. 2024Evaluating the World Model Implicit in a Generative Modelcited by 113
  8. 2024Transformers represent belief state geometry in their residual streamcited by 77
  9. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 63
  10. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 62
  11. 2024Do language models plan ahead for future tokens?cited by 49
  12. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  13. 2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 34
  14. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 30
  15. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  16. 2024Tracking the Feature Dynamics in LLM Training: A Mechanistic Studycited by 16
  17. 2025Hidden Breakthroughs in Language Model Trainingcited by 15
  18. 2024The Belief State Transformercited by 12
  19. 2025Revisiting the Othello World Model Hypothesiscited by 11
  20. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  21. 2025Looking beyond the next tokencited by 5
  22. 2025Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Auto-Regressive Next-Token Predictors are Universal Learners
  2. 2026From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks
  3. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
  4. 2025Emergent Response Planning in LLMs
  5. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
  6. 2025From Next Token Prediction to (STRIPS) World Models
  7. 2025Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning
  8. 2026A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases