# Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models

arXiv: https://arxiv.org/abs/2504.04264 · HTML: https://mlsanity.com/paper/2504.04264/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 5 Apr 2025.

## Builds on (most cited first, 20 of 20)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons](https://mlsanity.com/paper/2308.13198/) (2023)
- [Language Anisotropic Cross-Lingual Model Editing](https://mlsanity.com/paper/2205.12677/) (2022)
- [Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons](https://mlsanity.com/paper/2408.03247/) (2024)
- [BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning](https://mlsanity.com/paper/2406.17764/) (2024)

## Built on by (most cited first, 30 of 44)

- [Refusal Direction is Universal Across Safety-Aligned Languages](https://mlsanity.com/paper/2505.17306/) (2025)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes](https://mlsanity.com/paper/2505.14815/) (2025)
- [Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline](https://mlsanity.com/paper/2505.20546/) (2025)
- [Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs](https://mlsanity.com/paper/2504.09378/) (2025)
- [Tracing Multilingual Factual Knowledge Acquisition in Pretraining](https://mlsanity.com/paper/2505.14824/) (2025)
- [The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failure](https://mlsanity.com/paper/2506.22724/) (2025)
- [Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models](https://mlsanity.com/paper/2505.16538/) (2025)
- [A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages](https://mlsanity.com/paper/2510.09555/) (2025)
- [Different types of syntactic agreement recruit the same units within large language models](https://mlsanity.com/paper/2512.03676/) (2025)
- [Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG](https://mlsanity.com/paper/2509.13930/) (2025)
- [Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?](https://mlsanity.com/paper/2507.12838/) (2025)
- [On the Entity-Level Alignment in Crosslingual Consistency](https://mlsanity.com/paper/2510.10280/) (2025)
- [Rethinking Cross-lingual Gaps from a Statistical Viewpoint](https://mlsanity.com/paper/2510.15551/) (2025)
- [Do LLM hallucination detectors suffer from low-resource effect?](https://mlsanity.com/paper/2601.16766/) (2026)
- [The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languages](https://mlsanity.com/paper/2606.23165/) (2026)
- [Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics](https://mlsanity.com/paper/2508.11017/) (2025)
- [Post-Training Language Models for Crosslingual Consistency](https://mlsanity.com/paper/2603.04678/) (2026)
- [Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models](https://mlsanity.com/paper/2511.07498/) (2025)
- [GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation](https://mlsanity.com/paper/2604.24929/) (2026)
- [Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility](https://mlsanity.com/paper/2606.03291/) (2026)
- [When Languages Disagree: Self-Evolving Multilingual LLM Judges](https://mlsanity.com/paper/2606.08092/) (2026)
- [Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops](https://mlsanity.com/paper/2608.28860/) (2026)
- [Neuron-Level Analysis of Cultural Understanding in Large Language Models](https://mlsanity.com/paper/2510.08284/) (2025)
- [A Diagnostic Benchmark for Sweden-Related Factual Knowledge](https://mlsanity.com/paper/2510.21360/) (2025)
- [Evaluating Contextually Mediated Factual Recall in Multilingual Large Language Models](https://mlsanity.com/paper/2601.12555/) (2026)
- [Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation](https://mlsanity.com/paper/2605.12515/) (2026)
- [Learning to Route Languages for Multilingual Policy Optimization](https://mlsanity.com/paper/2605.25360/) (2026)
- [Cross-Lingual Steering for Figurative Language Generation](https://mlsanity.com/paper/2605.30443/) (2026)

## Abstract (arXiv)

Multilingual language models (MLMs) store factual knowledge across languages but often struggle to provide consistent responses to semantically equivalent prompts in different languages. While previous studies point out this cross-lingual inconsistency issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in a language-independent concept space through most layers, and only transition to language-specific spaces in the final layers. Failures during the language transition often result in incorrect predictions in the target language, even when the answers are correct in other languages. To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Our findings shed light on the internal mechanisms of MLMs and provide a lightweight, effective strategy for producing more consistent factual outputs.