# Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection

arXiv: https://arxiv.org/abs/2605.23036 · HTML: https://mlsanity.com/paper/2605.23036/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 23 of 23)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering](https://mlsanity.com/paper/2410.15999/) (2024)
- [Steering Large Language Model Activations in Sparse Spaces](https://mlsanity.com/paper/2503.00177/) (2025)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)
- [The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling](https://mlsanity.com/paper/2410.09223/) (2024)
- [Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation](https://mlsanity.com/paper/2404.11201/) (2024)
- [Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation](https://mlsanity.com/paper/2507.22608/) (2025)
- [Causal Language Control in Multilingual Transformers via Sparse Feature Steering](https://mlsanity.com/paper/2507.13410/) (2025)
- [CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark](https://mlsanity.com/paper/2601.08331/) (2026)
- [Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinement](https://mlsanity.com/paper/2509.23799/) (2025)
- [Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering](https://mlsanity.com/paper/2505.15038/) (2025)
- [Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons](https://mlsanity.com/paper/2506.01629/) (2025)

## Built on by (most cited first, 4 of 4)

- [Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation](https://mlsanity.com/paper/2606.18389/) (2026)
- [From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop](https://mlsanity.com/paper/2608.11171/) (2026)
- [Tracing Stereotypes from Representation to Output in Multilingual LLMs](https://mlsanity.com/paper/2609.08322/) (2026)
- [Semantic Fibers and Cross-Gram Interference: A Calculus of Safety Drift in Overcomplete Representations](https://mlsanity.com/paper/2609.14861/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) enable feature-level mechanistic interpretability and activation steering in large language models (LLMs), but SAE-based language control remains unreliable in multilingual settings: most SAEs are trained on English-only data, and steering layers are chosen heuristically. We address these limitations by advancing a principled, mechanistic account of multilingual language steering with SAEs. First, we show that training SAEs on multilingual data consistently strengthens cross-lingual representations and yields more reliable, quality-preserving language control across layers and model families. Second, we introduce an a priori steering layer-selection rule based on the intersection of multilingual alignment and language separability, which predicts effective intervention depths without exhaustive layerwise search. We evaluate our approach on LLaMA-3.1-8B and Gemma-2-9B across machine translation and cross-lingual summarization (CrossSumm), using SpBLEU, ROUGE-L, COMET, and LaSE. Our results show that multilingual SAEs combined with intersection-selected layers stabilize the trade-off between language identification accuracy and generation quality, providing a principled, predictive, representation-level account of multilingual SAE steering.