mlsanityData to 14 Oct 2026

Evolution of Concepts in Language Model Pre-Training

arXiv 2509.17196 ↗First posted 21 Sep 2025cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Language models obtain extensive capabilities through pre-training. However, the pre-training process remains a black box. In this work, we track linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders. We find that most features begin to form around a specific point, while more complex patterns emerge in later training stages. Feature attribution analyses reveal causal connections between feature evolution and downstream performance. Our feature-level observations are highly consistent with previous findings on Transformer's two-stage learning process, which we term a statistical learning phase and a feature learning phase. Our work opens up the possibility to track fine-grained representation progress during language model learning dynamics.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 52 of its 71 references that are in the map. First posted on arXiv 21 Sep 2025.

Builds on

52 of 71 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  12. 2016Understanding intermediate layers using linear classifier probescited by 1,353
38 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  3. 2022Toy Models of Superpositioncited by 971
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  8. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  9. 2024Scaling and evaluating sparse autoencoderscited by 553
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  11. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  12. 2021Explaining Neural Scaling Lawscited by 363
  13. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  14. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
  15. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  16. 2018Training behavior of deep neural network in frequency domaincited by 174
  17. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  18. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  19. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  20. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  21. 2024Universal Neurons in GPT2 Language Modelscited by 102
  22. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  23. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  24. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  25. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  26. 2020Tensor Programs III: Neural Matrix Lawscited by 49
  27. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  28. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  29. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 35
  30. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  31. 2024Transformers Can Represent n-gram Language Modelscited by 32
  32. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 31
  33. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  34. 2024Understanding Transformers via N-gram Statisticscited by 26
  35. 2020Exploring the Linear Subspace Hypothesis in Gender Bias Mitigationcited by 25
  36. 2024Automatically Identifying Local and Global Circuits with Linear Computation Graphscited by 24
  37. 2017Do Neural Nets Learn Statistical Laws behind Natural Language?cited by 16
  38. 2024Estimating the Local Learning Coefficient at Scalecited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  2. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
  3. 2026Learning is Forgetting: LLM Training As Lossy Compressioncited by 5
  4. 2026Convergent World Representations and Divergent Taskscited by 2
  5. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustnesscited by 1
  6. 2026fmxcoders: Factorized Masked Crosscoders for Cross-Layer Feature Discoverycited by 0
  7. 2026From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Modelscited by 0
  8. 2026Features have life history. And we should carecited by 0