mlsanityData to 14 Oct 2026

Tracking the Feature Dynamics in LLM Training: A Mechanistic Study

arXiv 2412.17626 ↗First posted 23 Dec 2024cs.LG cs.CLcited by 16

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Understanding training dynamics and feature evolution is crucial for the mechanistic interpretability of large language models (LLMs). Although sparse autoencoders (SAEs) have been used to identify features within LLMs, a clear picture of how these features evolve during training remains elusive. In this study, we (1) introduce SAE-Track, a novel method for efficiently obtaining a continual series of SAEs, providing the foundation for a mechanistic study that covers (2) the semantic evolution of features, (3) the underlying processes of feature formation, and (4) the directional drift of feature vectors. Our work provides new insights into the dynamics of features in LLMs, enhancing our understanding of training mechanisms and feature evolution. For reproducibility, our code is available at https://github.com/Superposition09m/SAE-Track.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 19 of its 33 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

19 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Features4 Nov
  2. 2024Applying sparse autoencoders to unlearn knowledge in language models25 Oct
  3. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 15

  1. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2022Toy Models of Superpositioncited by 971
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  9. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  10. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  11. 2024Mechanistic Permutability: Match Features Across Layerscited by 22
  12. 2024Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Modelscited by 18
3 more
  1. 2024Towards the Dynamics of a DNN Learning Symbolic Interactionscited by 13
  2. 2024Constrain Alignment with Sparse Autoencoderscited by 12
  3. 2021Trap of Feature Diversity in the Learning of MLPscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  2. 2025Hidden Dynamics of Massive Activations in Transformer Trainingcited by 10
  3. 2026G-Drift MIA: Membership Inference via Gradient-Induced Feature Drift in LLMscited by 8
  4. 2025How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoderscited by 5
  5. 2025DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spacescited by 2
  6. 2025The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Modelscited by 1
  7. 2025A Review of Developmental Interpretability in Large Language Modelscited by 1
  8. 2026How Do Large Language Models Learn Concepts During Continual Pre-Training?cited by 1
  9. 2026When Do Concepts Become Functionally Sufficient During Language-Model Training?cited by 1
  10. 2026Capsule Lens: Locating and Tracking Concept Geometry in Model Representationscited by 1
  11. 2025TopK Language Modelscited by 0
  12. 2026Understanding the Emergence of Seemingly Useless Features in Next-Token Predictorscited by 0
4 more
  1. 2026Features have life history. And we should carecited by 0
  2. 2026Feature Evolution and Migration during Vision Transformer Trainingcited by 0
  3. 2026Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Modelscited by 0
  4. 2026Tracing Morphology, Semantics, and Divergence in Language Acquisitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  2. 2024Evolution of SAE Features Across Layers in LLMs
  3. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
  4. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders
  5. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
  6. 2026Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations
  7. 2026Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders
  8. 2026SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training