mlsanityData to 14 Oct 2026

SharedSAE: One Feature Dictionary Across Language Models

arXiv 2609.04344 ↗First posted 3 Sep 2026cs.LG cs.CLcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 31 of its 60 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

31 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet28 May
  2. 2025SAEs Are Good for Steering -- If You Select the Right Features26 May
  3. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability12 Mar
  4. 2025FADE: Why Bad Descriptions Happen to Good Features24 Feb
  5. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders9 Oct
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  4. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  6. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  7. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  8. 2014Understanding image representations by measuring their equivariance and equivalencecited by 243
  9. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  10. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  11. 2022Relative representations enable zero-shot latent space communicationcited by 191
  12. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 86
13 more
  1. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  2. 2023Rosetta Neurons: Mining the Common Units in a Model Zoocited by 46
  3. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  4. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  5. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  6. 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scalecited by 16
  7. 2025Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoderscited by 14
  8. 2026Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMscited by 12
  9. 2026SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Datacited by 11
  10. 2025SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretabilitycited by 9
  11. 2025Atlas-Alignment: Making Interpretability Transferable Across Language Modelscited by 3
  12. 2026One Adapter Pair per Model: A Universal Activation Interface for Language Modelscited by 2
  13. 2026Reference Feature Atlases for Mechanistic Auditing of Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
  2. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features
  3. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
  4. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
  6. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  7. 2025Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders
  8. 2025FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies