mlsanityData to 14 Oct 2026

Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders

arXiv 2505.05111 ↗First posted 8 May 2025cs.CLcited by 13

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

The mechanisms behind multilingual capabilities in Large Language Models (LLMs) have been examined using neuron-based or internal-activation-based methods. However, these methods often face challenges such as superposition and layer-wise activation variance, which limit their reliability. Sparse Autoencoders (SAEs) offer a more nuanced analysis by decomposing the activations of LLMs into a sparse linear combination of SAE features. We introduce a novel metric to assess the monolinguality of features obtained from SAEs, discovering that some features are strongly related to specific languages. Additionally, we show that ablating these SAE features only significantly reduces abilities in one language of LLMs, leaving others almost unaffected. Interestingly, we find some languages have multiple synergistic SAE features, and ablating them together yields greater improvement than ablating individually. Moreover, we leverage these SAE-derived language-specific features to enhance steering vectors, achieving control over the language generated by LLMs. The code is publicly available at https://github.com/Aatrox103/multilingual-llm-features.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 23 of its 36 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

23 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Can sparse autoencoders be used to decompose and interpret steering vectors?13 Nov
  2. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
  3. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  8. 2023Steering Language Models With Activation Engineeringcited by 922
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  11. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  12. 2024How do Large Language Models Handle Multilingualism?cited by 150
6 more
  1. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
  2. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  3. 2023An Open Dataset and Model for Language Identificationcited by 50
  4. 2025Personalized Generation In Large Model Era: A Surveycited by 45
  5. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?cited by 41
  6. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  2. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architecturescited by 14
  3. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 10
  4. 2026Learning Retrieval Models with Sparse Autoencoderscited by 6
  5. 2025What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representationscited by 5
  6. 2025SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMscited by 2
  7. 2026AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMscited by 2
  8. 2026Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Headscited by 1
  9. 2026LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steeringcited by 1
  10. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  11. 2026Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languagescited by 0
  12. 2026Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoderscited by 0
1 more
  1. 2026Language Models Have Interpretable Narrative Mechanismscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
  2. 2025Causal Language Control in Multilingual Transformers via Sparse Feature Steering
  3. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  4. 2026Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference
  5. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
  6. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
  7. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders
  8. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features