mlsanityData to 14 Oct 2026

Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference

arXiv 2608.04904 ↗First posted 5 Aug 2026cs.CLcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show macro-average accuracy improvements of 11.28 percentage points on XCOPA, 4.11 points on XNLI, and 7.84 points on PAWS-X. Additional evaluation with Qwen2.5-7B-Instruct yields a macro-average improvement of 3.33 percentage points on XCOPA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 23 of its 33 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

23 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Causal Language Control in Multilingual Transformers via Sparse Feature Steering17 Jul
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2023Steering Language Models With Activation Engineeringcited by 922
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  11. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  12. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
9 more
  1. 2022The Geometry of Multilingual Language Model Representationscited by 95
  2. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 85
  3. 2025Do Multilingual LLMs Think In English?cited by 78
  4. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  5. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  6. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  7. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 19
  8. 2025ReCoVeR the Target Language: Language Steering without Sacrificing Task Performancecited by 6
  9. 2026Language Steering for Multilingual In-Context Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
  2. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
  3. 2026LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
  4. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
  5. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
  6. 2026Cross-Lingual Activation Steering for Multilingual Language Models
  7. 2025Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
  8. 2026Enhancing SAE-based Steering via Neighbor Integrated Feature Selection