# Pretrained Language Model Embryology: The Birth of ALBERT

arXiv: https://arxiv.org/abs/2010.02480 · HTML: https://mlsanity.com/paper/2010.02480/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 30 of 30)

- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [The MultiBERTs: BERT Reproductions for Robustness Analysis](https://mlsanity.com/paper/2106.16163/) (2021)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Probing Across Time: What Does RoBERTa Know and When?](https://mlsanity.com/paper/2104.07885/) (2021)
- [Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability](https://mlsanity.com/paper/2308.15419/) (2023)
- [Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models](https://mlsanity.com/paper/2205.11758/) (2022)
- [Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models](https://mlsanity.com/paper/2310.16570/) (2023)
- [An In-depth Survey of Large Language Model-based Artificial Intelligence Agents](https://mlsanity.com/paper/2309.14365/) (2023)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization](https://mlsanity.com/paper/2203.10945/) (2022)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training](https://mlsanity.com/paper/2310.16484/) (2023)
- [Interpreting Language Models Through Knowledge Graph Extraction](https://mlsanity.com/paper/2111.08546/) (2021)
- [Lexical Generalization Improves with Larger Models and Longer Training](https://mlsanity.com/paper/2210.12673/) (2022)
- [Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Information](https://mlsanity.com/paper/2205.03759/) (2022)
- [Training Dynamics of Contextual N-Grams in Language Models](https://mlsanity.com/paper/2311.00863/) (2023)
- [IIITT@LT-EDI-EACL2021-Hope Speech Detection: There is always Hope in Transformers](https://mlsanity.com/paper/2104.09066/) (2021)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model](https://mlsanity.com/paper/2310.15477/) (2023)
- [VIRL: Volume-Informed Representation Learning towards Few-shot Manufacturability Estimation](https://mlsanity.com/paper/2406.12286/) (2024)
- [Is neural language acquisition similar to natural? A chronological probing study](https://mlsanity.com/paper/2207.00560/) (2022)
- [From Text to Emoji: How PEFT-Driven Personality Manipulation Unleashes the Emoji Potential in LLMs](https://mlsanity.com/paper/2409.10245/) (2024)
- [Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings](https://mlsanity.com/paper/2510.19694/) (2025)
- [The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages](https://mlsanity.com/paper/2511.09197/) (2025)
- [Gender Biases Unexpectedly Fluctuate in the Pre-training Stage of Masked Language Models](https://mlsanity.com/paper/2211.14639/) (2022)
- [Fairness Dynamics During Training](https://mlsanity.com/paper/2506.01709/) (2025)
- [Plausibility Vaccine: Injecting LLM Knowledge for Event Plausibility](https://mlsanity.com/paper/2503.12667/) (2025)
- [Domain Pre-training Impact on Representations](https://mlsanity.com/paper/2505.24455/) (2025)
- [Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models](https://mlsanity.com/paper/2506.03434/) (2025)

## Abstract (arXiv)

While behaviors of pretrained language models (LMs) have been thoroughly examined, what happened during pretraining is rarely studied. We thus investigate the developmental process from a set of randomly initialized parameters to a totipotent language model, which we refer to as the embryology of a pretrained language model. Our results show that ALBERT learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. We also find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks' performance. These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge. We will provide source codes and pretrained models to reproduce our results at https://github.com/d223302/albert-embryology.