# Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders arXiv: https://arxiv.org/abs/2505.05111 ยท HTML: https://mlsanity.com/paper/2505.05111/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. - Builds on 23 of its 36 references that are in the map. - First posted on arXiv 8 May 2025. ## Placed in - Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/) ## Builds on (most cited first, 23 of 23) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024) - [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024) - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023) - [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022) - [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023) - [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024) - [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023) - [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022) - [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021) - [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024) - [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024) - [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024) - [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024) - [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024) - [Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models](https://mlsanity.com/paper/2411.14257/) (2024) - [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024) - [An Open Dataset and Model for Language Identification](https://mlsanity.com/paper/2305.13820/) (2023) - [Personalized Generation In Large Model Era: A Survey](https://mlsanity.com/paper/2503.02614/) (2025) - [Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?](https://mlsanity.com/paper/2408.10811/) (2024) - [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022) - [Can sparse autoencoders be used to decompose and interpret steering vectors?](https://mlsanity.com/paper/2411.08790/) (2024) ## Built on by (most cited first, 13 of 13) - [Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models](https://mlsanity.com/paper/2605.11887/) (2026) - [Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures](https://mlsanity.com/paper/2506.01197/) (2025) - [Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents](https://mlsanity.com/paper/2510.04491/) (2025) - [Learning Retrieval Models with Sparse Autoencoders](https://mlsanity.com/paper/2603.13277/) (2026) - [What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations](https://mlsanity.com/paper/2507.20279/) (2025) - [SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs](https://mlsanity.com/paper/2507.14894/) (2025) - [AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs](https://mlsanity.com/paper/2601.22710/) (2026) - [Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads](https://mlsanity.com/paper/2603.05772/) (2026) - [LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering](https://mlsanity.com/paper/2604.03532/) (2026) - [Steered Generation via Gradient-Based Optimization on Sparse Query Features](https://mlsanity.com/paper/2605.23040/) (2026) - [Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages](https://mlsanity.com/paper/2608.18545/) (2026) - [Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders](https://mlsanity.com/paper/2608.23809/) (2026) - [Language Models Have Interpretable Narrative Mechanisms](https://mlsanity.com/paper/2610.12989/) (2026) ## Abstract (arXiv) The mechanisms behind multilingual capabilities in Large Language Models (LLMs) have been examined using neuron-based or internal-activation-based methods. However, these methods often face challenges such as superposition and layer-wise activation variance, which limit their reliability. Sparse Autoencoders (SAEs) offer a more nuanced analysis by decomposing the activations of LLMs into a sparse linear combination of SAE features. We introduce a novel metric to assess the monolinguality of features obtained from SAEs, discovering that some features are strongly related to specific languages. Additionally, we show that ablating these SAE features only significantly reduces abilities in one language of LLMs, leaving others almost unaffected. Interestingly, we find some languages have multiple synergistic SAE features, and ablating them together yields greater improvement than ablating individually. Moreover, we leverage these SAE-derived language-specific features to enhance steering vectors, achieving control over the language generated by LLMs. The code is publicly available at https://github.com/Aatrox103/multilingual-llm-features.