# Transferring Linear Features Across Language Models With Model Stitching

arXiv: https://arxiv.org/abs/2506.06609 · HTML: https://mlsanity.com/paper/2506.06609/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 48 of its 67 references that are in the map.
- First posted on arXiv 7 Jun 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 30 of 48)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)

## Built on by (most cited first, 21 of 21)

- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse](https://mlsanity.com/paper/2608.03893/) (2026)
- [Latent Cache Flow: Model-to-Model Communication Without Text](https://mlsanity.com/paper/2605.22863/) (2026)
- [A Negative Result on Cross-Model Activation Transfer in a Pythia Multi-Hop Setting](https://mlsanity.com/paper/2606.03280/) (2026)
- [When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs](https://mlsanity.com/paper/2608.04893/) (2026)
- [Frozen Memory Is Not Enough: Rethinking External Memory as Extraction](https://mlsanity.com/paper/2608.17050/) (2026)
- [Align Then Adapt: Label-Efficient Adapter Learning for Asymmetric Dense Retrieval](https://mlsanity.com/paper/2604.03403/) (2026)
- [Do Models Share Safety Representations? Cross-Model Steering for Safe Visual Generation](https://mlsanity.com/paper/2606.05290/) (2026)
- [Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations](https://mlsanity.com/paper/2607.20379/) (2026)
- [One Adapter Pair per Model: A Universal Activation Interface for Language Models](https://mlsanity.com/paper/2608.09521/) (2026)
- [ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Data](https://mlsanity.com/paper/2604.17663/) (2026)
- [Deep Minds and Shallow Probes](https://mlsanity.com/paper/2605.11448/) (2026)
- [Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m](https://mlsanity.com/paper/2605.24577/) (2026)
- [Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures](https://mlsanity.com/paper/2606.16897/) (2026)
- [How Far Do Simple Transformations Translate Across Text Embedding Models?](https://mlsanity.com/paper/2608.05980/) (2026)
- [SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization](https://mlsanity.com/paper/2608.13538/) (2026)
- [Approximate Homomorphisms and Convergent Representations in Transducers](https://mlsanity.com/paper/2608.20428/) (2026)
- [A Survey on the Linear Representation Hypothesis](https://mlsanity.com/paper/2609.22695/) (2026)
- [SuperAI: Merging Capabilities Across Heterogeneous Foundation Models](https://mlsanity.com/paper/2610.15322/) (2026)
- [CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoff](https://mlsanity.com/paper/2610.15509/) (2026)
- [Share the Symbol, Not the Vector: Training-Free Fact Communication Across Language Models](https://mlsanity.com/paper/2610.19696/) (2026)

## Abstract (arXiv)

In this work, we demonstrate that affine mappings between residual streams of language models is a cheap way to effectively transfer represented features between models. We apply this technique to transfer the weights of Sparse Autoencoders (SAEs) between models of different sizes to compare their representations. We find that small and large models learn similar representation spaces, which motivates training expensive components like SAEs on a smaller model and transferring to a larger model at a FLOPs savings. In particular, using a small-to-large transferred SAE as initialization can lead to 50% cheaper training runs when training SAEs on larger models. Next, we show that transferred probes and steering vectors can effectively recover ground truth performance. Finally, we dive deeper into feature-level transferability, finding that semantic and structural features transfer noticeably differently while specific classes of functional features have their roles faithfully mapped. Overall, our findings illustrate similarities and differences in the linear representation spaces of small and large models and demonstrate a method for improving the training efficiency of SAEs.