# Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models

arXiv: https://arxiv.org/abs/2506.00653 · HTML: https://mlsanity.com/paper/2506.00653/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 29 of 29)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Knowledge Fusion of Large Language Models](https://mlsanity.com/paper/2401.10491/) (2024)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Similarity and Matching of Neural Network Representations](https://mlsanity.com/paper/2110.14633/) (2021)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://mlsanity.com/paper/2402.12030/) (2024)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching](https://mlsanity.com/paper/2503.20083/) (2025)
- [ReSi: A Comprehensive Benchmark for Representational Similarity Measures](https://mlsanity.com/paper/2408.00531/) (2024)
- [Activation Space Interventions Can Be Transferred Between Large Language Models](https://mlsanity.com/paper/2503.04429/) (2025)
- [Towards Measuring Representational Similarity of Large Language Models](https://mlsanity.com/paper/2312.02730/) (2023)

## Built on by (most cited first, 7 of 7)

- [Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse](https://mlsanity.com/paper/2608.03893/) (2026)
- [Do Models Share Safety Representations? Cross-Model Steering for Safe Visual Generation](https://mlsanity.com/paper/2606.05290/) (2026)
- [The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment](https://mlsanity.com/paper/2604.06377/) (2026)
- [DreamReader: An Interpretability Toolkit for Text-to-Image Models](https://mlsanity.com/paper/2603.13299/) (2026)
- [Multilingual Embedding Probes Fail to Generalize Across Learner Corpora](https://mlsanity.com/paper/2604.07095/) (2026)
- [HyperTransport: Amortized Conditioning of T2I Generative Models](https://mlsanity.com/paper/2605.08254/) (2026)
- [Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps](https://mlsanity.com/paper/2610.09058/) (2026)

## Abstract (arXiv)

It has been hypothesized that neural networks with similar architectures trained on similar data learn shared representations relevant to the learning task. We build on this idea by extending the conceptual framework where representations learned across models trained on the same data can be expressed as linear combinations of a universal set of basis features. These basis features underlie the learning task itself and remain consistent across models, regardless of scale. From this framework, we propose the Linear Representation Transferability (LRT) Hypothesis -- that there exists an affine transformation between the representation spaces of different models. To test this hypothesis, we learn affine mappings between the hidden states of models of different sizes and evaluate whether steering vectors -- directions in hidden state space associated with specific model behaviors -- retain their semantic effect when transferred from small to large language models using the learned mappings. We find strong empirical evidence that such affine mappings can preserve steering behaviors. These findings suggest that representations learned by small models can be used to guide the behavior of large models, and that the LRT hypothesis may be a promising direction on understanding representation alignment across model scales.