# Tracking the Feature Dynamics in LLM Training: A Mechanistic Study

arXiv: https://arxiv.org/abs/2412.17626 · HTML: https://mlsanity.com/paper/2412.17626/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 19 of 19)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding](https://mlsanity.com/paper/2303.04245/) (2023)
- [Applying sparse autoencoders to unlearn knowledge in language models](https://mlsanity.com/paper/2410.19278/) (2024)
- [The Geometry of Concepts: Sparse Autoencoder Feature Structure](https://mlsanity.com/paper/2410.19750/) (2024)
- [Mechanistic Permutability: Match Features Across Layers](https://mlsanity.com/paper/2410.07656/) (2024)
- [Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models](https://mlsanity.com/paper/2402.19465/) (2024)
- [Towards the Dynamics of a DNN Learning Symbolic Interactions](https://mlsanity.com/paper/2407.19198/) (2024)
- [Constrain Alignment with Sparse Autoencoders](https://mlsanity.com/paper/2411.07618/) (2024)
- [Trap of Feature Diversity in the Learning of MLPs](https://mlsanity.com/paper/2112.00980/) (2021)

## Built on by (most cited first, 16 of 16)

- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Hidden Dynamics of Massive Activations in Transformer Training](https://mlsanity.com/paper/2508.03616/) (2025)
- [G-Drift MIA: Membership Inference via Gradient-Induced Feature Drift in LLMs](https://mlsanity.com/paper/2604.00419/) (2026)
- [How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders](https://mlsanity.com/paper/2503.06394/) (2025)
- [DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spaces](https://mlsanity.com/paper/2505.18441/) (2025)
- [The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models](https://mlsanity.com/paper/2505.19440/) (2025)
- [A Review of Developmental Interpretability in Large Language Models](https://mlsanity.com/paper/2508.15841/) (2025)
- [How Do Large Language Models Learn Concepts During Continual Pre-Training?](https://mlsanity.com/paper/2601.03570/) (2026)
- [When Do Concepts Become Functionally Sufficient During Language-Model Training?](https://mlsanity.com/paper/2608.15323/) (2026)
- [Capsule Lens: Locating and Tracking Concept Geometry in Model Representations](https://mlsanity.com/paper/2609.05575/) (2026)
- [TopK Language Models](https://mlsanity.com/paper/2506.21468/) (2025)
- [Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors](https://mlsanity.com/paper/2603.14087/) (2026)
- [Features have life history. And we should care](https://mlsanity.com/paper/2605.18789/) (2026)
- [Feature Evolution and Migration during Vision Transformer Training](https://mlsanity.com/paper/2608.20134/) (2026)
- [Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Models](https://mlsanity.com/paper/2609.21113/) (2026)
- [Tracing Morphology, Semantics, and Divergence in Language Acquisition](https://mlsanity.com/paper/2610.19041/) (2026)

## Abstract (arXiv)

Understanding training dynamics and feature evolution is crucial for the mechanistic interpretability of large language models (LLMs). Although sparse autoencoders (SAEs) have been used to identify features within LLMs, a clear picture of how these features evolve during training remains elusive. In this study, we (1) introduce SAE-Track, a novel method for efficiently obtaining a continual series of SAEs, providing the foundation for a mechanistic study that covers (2) the semantic evolution of features, (3) the underlying processes of feature formation, and (4) the directional drift of feature vectors. Our work provides new insights into the dynamics of features in LLMs, enhancing our understanding of training mechanisms and feature evolution. For reproducibility, our code is available at https://github.com/Superposition09m/SAE-Track.