mlsanityData to 14 Oct 2026

Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale

arXiv 2510.24963 ↗First posted 28 Oct 2025cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregressive language models exhibit highly consistent patterns of change in their behavior over the course of pretraining. Based on our analysis of over 1,400 language model checkpoints on over 110,000 tokens of English, we find that up to 98% of the variance in language model behavior at the word level can be explained by three simple heuristics: the unigram probability (frequency) of a given word, the n-gram probability of the word, and the semantic similarity between the word and its context. Furthermore, we see consistent behavioral phases in all language models, with their predicted probabilities for words overfitting to those words' n-gram probabilities for increasing n over the course of training. Taken together, these results suggest that learning in neural language models may follow a similar trajectory irrespective of model details.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 75 of its 105 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

75 of 105 references.

References · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  10. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  13. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  14. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  15. 20242 OLMo 2 Furiouscited by 340
  16. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  17. 2018Learning Word Vectors for 157 Languagescited by 280
  18. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  19. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  20. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  21. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  22. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  23. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  24. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  25. 2015Visualizing and Understanding Recurrent Networkscited by 160
  26. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  27. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  28. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  29. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  30. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  31. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
  32. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  33. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  34. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  35. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  36. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  37. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  38. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 54

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
  2. 2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 7
  3. 2026Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMscited by 3
  4. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 3
  5. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 2
  6. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  7. 2026Tracking the emergence of linguistic structure in self-supervised models learning from speechcited by 1
  8. 2026Smooth Scaling Laws Hide Stepwise Token Learningcited by 1
  9. 2026On the scaling relationship between cloze probabilities and language model next-token predictioncited by 0
  10. 2026Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Modelscited by 0
  11. 2026Beetle: A Bilingual Model Suite for Modelling Second-Language Processingcited by 0
  12. 2026What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetitioncited by 0
1 more
  1. 2026Lasting Effects of Abstract Pretraining Beyond Perplexitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Observational Scaling Laws and the Predictability of Language Model Performance
  2. 2024What Makes Two Language Models Think Alike?
  3. 2026Position: It's Time to Optimize LLMs for Self-Consistency
  4. 2024Improving the Robustness of Large Language Models via Consistency Alignment
  5. 2025A Probabilistic Perspective on Model Collapse
  6. 2026Benchmark Shadows: Data Alignment, Parameter Footprints, and Generalization in Large Language Models
  7. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models
  8. 2024Revisiting the Superficial Alignment Hypothesis