mlsanityData to 7 Oct 2026

Language Steering for Multilingual In-Context Learning

arXiv 2602.02326 ↗First posted 2 Feb 2026cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

If large language models operate in a universal semantic space, then switching between languages should require only a simple activation offset. To test this, we take multilingual in-context learning as a case study, where few-shot demonstrations are provided in English but the test query is in a target language. We propose language vectors, computed as the mean activation difference between parallel source and target language examples at a particular layer, and added as an offset to hidden states at inference time to shift the model's internal representations toward the target language. We evaluate our method across three multilingual tasks spanning 19 languages and three models. Our results show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested, demonstrating that a simple activation offset is sufficient to redirect a model's language mode without any parameter updates. Beyond performance, the vectors encode interpretable linguistic structure, with closely related languages forming tight clusters and vectors transferring across tasks, suggesting that language identity occupies separable and structured directions in a model's activation space.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

33 of 39 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  6. 2022Toy Models of Superpositioncited by 928
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2023Steering Language Models With Activation Engineeringcited by 824
  9. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  11. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  12. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
21 more
  1. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  2. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  3. 2024How do Large Language Models Handle Multilingualism?cited by 148
  4. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  5. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  6. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  7. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  8. 2023Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observationscited by 58
  9. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  10. 2022Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extractioncited by 40
  11. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  12. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  13. 2021Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Trainingcited by 21
  14. 2025Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulationcited by 17
  15. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  16. 2025ReCoVeR the Target Language: Language Steering without Sacrificing Task Performancecited by 5
  17. 2025Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-trainingcited by 4
  18. 2025Language Surgery in Multilingual Large Language Modelscited by 4
  19. 2026Cross-Lingual Activation Steering for Multilingual Language Modelscited by 4
  20. 2025Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learningcited by 3
  21. 2025Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Code-Switching Reveals Language Anchoring in Multilingual LLMscited by 1
  2. 2026Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Dropscited by 1
  3. 2026Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inferencecited by 0