# Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale

arXiv: https://arxiv.org/abs/2510.24963 · HTML: https://mlsanity.com/paper/2510.24963/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 105 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 30 of 75)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)

## Built on by (most cited first, 14 of 14)

- [Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment](https://mlsanity.com/paper/2601.10160/) (2026)
- [What do Language Models Learn and When? The Implicit Curriculum Hypothesis](https://mlsanity.com/paper/2604.08510/) (2026)
- [Predicting the Emergence of Induction Heads in Language Model Pretraining](https://mlsanity.com/paper/2511.16893/) (2025)
- [Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs](https://mlsanity.com/paper/2602.16085/) (2026)
- [Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics](https://mlsanity.com/paper/2606.06533/) (2026)
- [N-gram-like Language Models Predict Naturalistic Reading Time Best](https://mlsanity.com/paper/2603.09872/) (2026)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [Tracking the emergence of linguistic structure in self-supervised models learning from speech](https://mlsanity.com/paper/2604.02043/) (2026)
- [Smooth Scaling Laws Hide Stepwise Token Learning](https://mlsanity.com/paper/2606.29858/) (2026)
- [On the scaling relationship between cloze probabilities and language model next-token prediction](https://mlsanity.com/paper/2602.17848/) (2026)
- [Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Models](https://mlsanity.com/paper/2606.28524/) (2026)
- [Beetle: A Bilingual Model Suite for Modelling Second-Language Processing](https://mlsanity.com/paper/2609.22633/) (2026)
- [What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetition](https://mlsanity.com/paper/2609.29507/) (2026)
- [Lasting Effects of Abstract Pretraining Beyond Perplexity](https://mlsanity.com/paper/2609.38764/) (2026)

## Abstract (arXiv)

We show that across architecture (Transformer vs. Mamba vs. RWKV), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregressive language models exhibit highly consistent patterns of change in their behavior over the course of pretraining. Based on our analysis of over 1,400 language model checkpoints on over 110,000 tokens of English, we find that up to 98% of the variance in language model behavior at the word level can be explained by three simple heuristics: the unigram probability (frequency) of a given word, the n-gram probability of the word, and the semantic similarity between the word and its context. Furthermore, we see consistent behavioral phases in all language models, with their predicted probabilities for words overfitting to those words' n-gram probabilities for increasing n over the course of training. Taken together, these results suggest that learning in neural language models may follow a similar trajectory irrespective of model details.