# Language Steering for Multilingual In-Context Learning

arXiv: https://arxiv.org/abs/2602.02326 · HTML: https://mlsanity.com/paper/2602.02326/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment](https://mlsanity.com/paper/2305.05940/) (2023)
- [Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations](https://mlsanity.com/paper/2310.20246/) (2023)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction](https://mlsanity.com/paper/2203.08308/) (2022)
- [PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning](https://mlsanity.com/paper/2311.08711/) (2023)
- [Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment](https://mlsanity.com/paper/2311.08089/) (2023)
- [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021)
- [Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation](https://mlsanity.com/paper/2507.22608/) (2025)
- [Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models](https://mlsanity.com/paper/2505.16538/) (2025)
- [ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance](https://mlsanity.com/paper/2509.14814/) (2025)
- [Cross-Lingual Activation Steering for Multilingual Language Models](https://mlsanity.com/paper/2601.16390/) (2026)
- [Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training](https://mlsanity.com/paper/2504.20484/) (2025)

## Built on by (most cited first, 3 of 3)

- [Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops](https://mlsanity.com/paper/2608.28860/) (2026)
- [Code-Switching Reveals Language Anchoring in Multilingual LLMs](https://mlsanity.com/paper/2606.19668/) (2026)
- [Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference](https://mlsanity.com/paper/2608.04904/) (2026)

## Abstract (arXiv)

If multilingual language model representations contain separable language-associated structure, then shifting between languages may require only simple activation-space interventions. To test this hypothesis, we study multilingual in-context learning (ICL) as a controlled setting: few-shot demonstrations are provided in English while the test query is in a target language, creating a clear cross-lingual transfer challenge. We propose language vectors, which are computed as mean activation differences between parallel source and target language examples at a particular layer and applied as inference-time offsets to the model's hidden states. Across three tasks spanning 19 languages and three models, our method improves over English few-shot prompting without any parameter updates. Beyond performance, the vectors encode interpretable linguistic structure: closely related languages form tight clusters, and vectors transfer effectively across tasks and languages, with closely related languages' vectors recovering more of the gain than distant ones. Together, these findings suggest that language occupies structured directions in multilingual activation space, and that inference-time interventions on English demonstrations can narrow the cross-lingual ICL gap.