# Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention

arXiv: https://arxiv.org/abs/2410.12462 · HTML: https://mlsanity.com/paper/2410.12462/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 30 of 33)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning](https://mlsanity.com/paper/2309.04766/) (2023)
- [Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://mlsanity.com/paper/1902.09492/) (2019)
- [Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning](https://mlsanity.com/paper/2106.06937/) (2021)
- [Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations](https://mlsanity.com/paper/2310.20246/) (2023)
- [Adapting Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2401.06468/) (2024)
- [Inducing Language-Agnostic Multilingual Representations](https://mlsanity.com/paper/2008.09112/) (2020)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations](https://mlsanity.com/paper/2308.14186/) (2023)
- [Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs](https://mlsanity.com/paper/2406.09265/) (2024)

## Built on by (most cited first, 24 of 24)

- [The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks](https://mlsanity.com/paper/2504.15521/) (2025)
- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs](https://mlsanity.com/paper/2504.09378/) (2025)
- [Improving Multilingual Language Models by Aligning Representations through Steering](https://mlsanity.com/paper/2505.12584/) (2025)
- [ExpertSteer: Intervening in LLMs through Expert Knowledge](https://mlsanity.com/paper/2505.12313/) (2025)
- [Language Surgery in Multilingual Large Language Models](https://mlsanity.com/paper/2506.12450/) (2025)
- [Rethinking Cross-lingual Gaps from a Statistical Viewpoint](https://mlsanity.com/paper/2510.15551/) (2025)
- [Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges](https://mlsanity.com/paper/2412.12686/) (2024)
- [Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models](https://mlsanity.com/paper/2510.07213/) (2025)
- [Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops](https://mlsanity.com/paper/2608.28860/) (2026)
- [The Effect of Language Diversity When Fine-Tuning Large Language Models for Translation](https://mlsanity.com/paper/2505.13090/) (2025)
- [Evolving and Executing Research Plans via Double-Loop Multi-Agent Collaboration](https://mlsanity.com/paper/2510.06761/) (2025)
- [Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration](https://mlsanity.com/paper/2601.02906/) (2026)
- [Multilinguality as Sense Adaptation](https://mlsanity.com/paper/2601.10310/) (2026)
- [SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics](https://mlsanity.com/paper/2601.15809/) (2026)
- [The Latin Substrate: How Language Models Represent and Mediate Script Choice](https://mlsanity.com/paper/2605.31363/) (2026)
- [xMIx: High-Performance Serving-Time Platform for Mechanistic Interpretability Apps](https://mlsanity.com/paper/2607.22595/) (2026)
- [Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference](https://mlsanity.com/paper/2608.04904/) (2026)
- [Cross-lingual Representation Learning via Centroid Intervention Fusion](https://mlsanity.com/paper/2608.26357/) (2026)
- [A Systematic Evaluation of Cross-Lingual Consistency Enhancement Methods in Multilingual Language Models](https://mlsanity.com/paper/2609.04409/) (2026)
- [Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control](https://mlsanity.com/paper/2609.16967/) (2026)
- [Why Pretraining Fails to Share Cross-Lingual Knowledge](https://mlsanity.com/paper/2609.19291/) (2026)
- [Fair Fact-Checking: Closing the Cross-Lingual Gap in LLM Factual Judgement with RoSh](https://mlsanity.com/paper/2609.34678/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities in natural language processing but exhibit significant performance gaps among different languages. Most existing approaches to address these disparities rely on pretraining or fine-tuning, which are resource-intensive. To overcome these limitations without incurring significant costs, we propose Inference-Time Cross-Lingual Intervention (INCLINE), a novel framework that enhances LLM performance on low-performing (source) languages by aligning their internal representations with those of high-performing (target) languages during inference. INCLINE initially learns alignment matrices using parallel sentences from source and target languages through a Least-Squares optimization, and then applies these matrices during inference to transform the low-performing language representations toward the high-performing language space. Extensive experiments on nine benchmarks with five LLMs demonstrate that INCLINE significantly improves performance across diverse tasks and languages, compared to recent strong baselines. Our analysis demonstrates that INCLINE is highly cost-effective and applicable to a wide range of applications. In addition, we release the code to foster research along this line: https://github.com/weixuan-wang123/INCLINE.