mlsanityData to 14 Oct 2026

A Mechanistic Investigation of Supervised Fine Tuning

arXiv 2605.11426 ↗First posted 12 May 2026cs.AIcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

The cosine similarity between a large language model's hidden activations before and after Supervised Fine-Tuning (SFT) remains very high. This, at first glance, suggests that SFT leaves the model's activation geometry largely undisturbed. However, projecting both sets of activations through a Sparse Autoencoder (SAE) pretrained on the base model reveals that the underlying sparse latents diverge significantly. We introduce a novel investigative pipeline which utilizes these pretrained SAEs as a high-resolution diagnostic tool to mechanistically investigate the drivers of this representational divergence. Through our analytical pipeline, we discover task-specific and layer-specific distributions of the precise semantic features that are systematically altered during supervised fine-tuning. We additionally identify a layer-wise update profile specific to safety alignment. All code, experimental scripts, and analysis files associated with this work are publicly available at: https://github.com/ruhzi/sae-investigation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 5 of its 15 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

5 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 4

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Layer-Wise Evolution of Representations in Fine-Tuned Transformers: Insights from Sparse AutoEncoders
  2. 2026Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning
  3. 2026Decomposing the Depth Profile of Fine-Tuning
  4. 2026Diversity in Large Language Models under Supervised Fine-Tuning
  5. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  6. 2026MoRFI: Monotonic Sparse Autoencoder Feature Identification
  7. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  8. 2026Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features