mlsanityData to 14 Oct 2026

Hidden Breakthroughs in Language Model Training

arXiv 2506.15872 ↗First posted 18 Jun 2025cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Loss curves are smooth during most of model training, so visible discontinuities stand out as possible conceptual breakthroughs. Studying these breakthroughs enables a deeper understanding of learning dynamics, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training but they are obscured by a loss metric that collapses all variation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank training subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and natural language tasks, showing that POLCA recovers clusters that represent interpretable breakthroughs in the model's capabilities. We demonstrate the promise of these hidden phase transitions as a tool for unsupervised interpretability.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

30 of 36 references.

References · 30

  1. 2022In-context Learning and Induction Headscited by 964
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  3. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  5. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  7. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  8. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  9. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  10. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  11. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  12. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
18 more
  1. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  2. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  3. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113
  4. 2017Critical Learning Periods in Deep Neural Networkscited by 79
  5. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  6. 2022Linear Connectivity Reveals Generalization Strategiescited by 56
  7. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  8. 2020Deep Learning is Singular, and That's Goodcited by 49
  9. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  10. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  11. 2024Does SGD really happen in tiny subspaces?cited by 36
  12. 2023Neuron to Graph: Interpreting Language Model Neurons at Scalecited by 27
  13. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  14. 2024The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Modelscited by 10
  15. 2024Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalizationcited by 10
  16. 2019LCA: Loss Change Allocation for Neural Network Trainingcited by 9
  17. 2022Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hexcited by 8
  18. 2023CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebracited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
  2. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  3. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 6
  4. 2025Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignmentcited by 5
  5. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 5
  6. 2025In-Context Algebracited by 5
  7. 2026From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretabilitycited by 4
  8. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 3
  9. 2026Emergent Capabilities Arise Randomly from Learning Sparse Attention Patternscited by 1
  10. 2025Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attractioncited by 0
  11. 2026Understanding the Emergence of Seemingly Useless Features in Next-Token Predictorscited by 0
  12. 2026Mechanistic Foundations of Goal-Directed Controlcited by 0
3 more
  1. 2026A Systematic Study of Behavioral Cloning for Scientific Data Annotationcited by 0
  2. 2026Neural Dynamics as the Composition of Quantized Unitscited by 0
  3. 2026Don't Forget! Decomposing the Training Dynamics of Memorization in Language Modelscited by 0