# Causal Language Control in Multilingual Transformers via Sparse Feature Steering

arXiv: https://arxiv.org/abs/2507.13410 · HTML: https://mlsanity.com/paper/2507.13410/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 6 of its 12 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 6 of 6)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)

## Built on by (most cited first, 12 of 12)

- [Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation](https://mlsanity.com/paper/2507.22608/) (2025)
- [EasySteer: A Unified Framework for High-Performance and Extensible LLM Steering](https://mlsanity.com/paper/2509.25175/) (2025)
- [CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark](https://mlsanity.com/paper/2601.08331/) (2026)
- [From Weights to Activations: Is Steering the Next Frontier of Adaptation?](https://mlsanity.com/paper/2604.14090/) (2026)
- [Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection](https://mlsanity.com/paper/2605.23036/) (2026)
- [Indic-TunedLens: Interpreting Multilingual Models in Indian Languages](https://mlsanity.com/paper/2602.15038/) (2026)
- [LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering](https://mlsanity.com/paper/2604.03532/) (2026)
- [Toward Mechanistic Interpretability of an AI Foundation Model Fine-Tuned for Atmospheric Chemistry](https://mlsanity.com/paper/2607.20778/) (2026)
- [Steered Generation via Gradient-Based Optimization on Sparse Query Features](https://mlsanity.com/paper/2605.23040/) (2026)
- [Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation](https://mlsanity.com/paper/2606.18389/) (2026)
- [Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference](https://mlsanity.com/paper/2608.04904/) (2026)
- [Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3](https://mlsanity.com/paper/2609.04808/) (2026)

## Abstract (arXiv)

Deterministically controlling the target generation language of large multilingual language models (LLMs) remains a fundamental challenge, particularly in zero-shot settings where neither explicit language prompts nor fine-tuning are available. In this work, we investigate whether sparse autoencoder (SAE) features, previously shown to correlate with interpretable model behaviors, can be leveraged to steer the generated language of LLMs during inference. Leveraging pretrained SAEs on the residual streams of Gemma-2B and Gemma-9B, we identify features whose activations differ most significantly between English and four target languages: Chinese, Japanese, Spanish, and French. By modifying just a single SAE feature at one transformer layer, we achieve controlled language shifts with up to 90% success, as measured by FastText language classification, while preserving semantic fidelity according to LaBSE (Language-Agnostic BERT Sentence Embedding) similarity. Our analysis reveals that language steering is most effective in mid-to-late transformer layers and is amplified by specific attention heads disproportionately associated with language-sensitive SAE features. These results demonstrate the promise of sparse feature steering as a lightweight and interpretable mechanism for controllable multilingual generation.