# Tracing Morphology, Semantics, and Divergence in Language Acquisition

arXiv: https://arxiv.org/abs/2610.19041 · HTML: https://mlsanity.com/paper/2610.19041/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Divergent Creativity in Humans and Large Language Models](https://mlsanity.com/paper/2405.13012/) (2024)
- [MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies](https://mlsanity.com/paper/2502.00894/) (2025)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [Theoretical Analysis of Byte-Pair Encoding](https://mlsanity.com/paper/2411.08671/) (2024)
- [Tracking the Feature Dynamics in LLM Training: A Mechanistic Study](https://mlsanity.com/paper/2412.17626/) (2024)
- [Transformer verbatim in-context retrieval across time and scale](https://mlsanity.com/paper/2411.07075/) (2024)
- [Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training](https://mlsanity.com/paper/2502.04066/) (2025)
- [IMPACT: Inflectional Morphology Probes Across Complex Typologies](https://mlsanity.com/paper/2506.23929/) (2025)
- [Weight Tying Biases Token Embeddings Towards the Output Space](https://mlsanity.com/paper/2603.26663/) (2026)

## Abstract (arXiv)

Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored. In this work, we leverage intermediate checkpoints from open-source models (e.g., Pythia and OLMo) under teacher-forcing evaluation to systematically investigate language acquisition across training steps and model scales. We focus on four core characteristics: morphological structure acquisition, embedding drift, repetition dominance, and co-occurrence effects. Our findings reveal that models acquire morphological structures well before semantic embeddings stabilize. Furthermore, we show that training dataset size has a positive impact on divergence recognition, while co-occurrence during pre-training and repetition during inference have a negative impact on divergence recognition and associative link generation.