# Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs

arXiv: https://arxiv.org/abs/2408.10646 · HTML: https://mlsanity.com/paper/2408.10646/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 24 of 24)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [How Do Large Language Models Acquire Factual Knowledge During Pretraining?](https://mlsanity.com/paper/2406.11813/) (2024)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons](https://mlsanity.com/paper/2404.02431/) (2024)
- [Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons](https://mlsanity.com/paper/2308.13198/) (2023)
- [Language Anisotropic Cross-Lingual Model Editing](https://mlsanity.com/paper/2205.12677/) (2022)
- [XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge](https://mlsanity.com/paper/2109.12573/) (2021)
- [Cross-lingual Editing in Multilingual Language Models](https://mlsanity.com/paper/2401.10521/) (2024)
- [How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning](https://mlsanity.com/paper/2305.13286/) (2023)
- [Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge](https://mlsanity.com/paper/2403.05189/) (2024)
- [MPN: Leveraging Multilingual Patch Neuron for Cross-lingual Model Editing](https://mlsanity.com/paper/2401.03190/) (2024)

## Built on by (most cited first, 21 of 21)

- [ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2502.21228/) (2025)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment](https://mlsanity.com/paper/2602.16660/) (2026)
- [Pretraining Language Models for Diachronic Linguistic Change Discovery](https://mlsanity.com/paper/2504.05523/) (2025)
- [Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale](https://mlsanity.com/paper/2509.14008/) (2025)
- [Time Travel Engine: A Shared Latent Chronological Manifold Enables Historical Navigation in Large Language Models](https://mlsanity.com/paper/2601.06437/) (2026)
- [Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops](https://mlsanity.com/paper/2608.28860/) (2026)
- [Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History](https://mlsanity.com/paper/2501.09154/) (2025)
- [Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation](https://mlsanity.com/paper/2605.12515/) (2026)
- [Position: It's Time to Optimize LLMs for Self-Consistency](https://mlsanity.com/paper/2608.05188/) (2026)
- [Skill Issue: Are Skills Language-Invariant in LLMs?](https://mlsanity.com/paper/2608.25832/) (2026)
- [Can Gradient Descent Simulate Prompting?](https://mlsanity.com/paper/2506.20989/) (2025)
- [Language models struggle with compartmentalization](https://mlsanity.com/paper/2605.19284/) (2026)
- [Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer](https://mlsanity.com/paper/2606.21954/) (2026)
- [When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs](https://mlsanity.com/paper/2607.21445/) (2026)
- [Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection](https://mlsanity.com/paper/2608.00207/) (2026)
- [CyrillicQA: The Influence of Phonetically Encoded Secret Language on LLM Performance](https://mlsanity.com/paper/2608.21462/) (2026)
- [Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders](https://mlsanity.com/paper/2608.23809/) (2026)
- [Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control](https://mlsanity.com/paper/2609.16967/) (2026)
- [Why Pretraining Fails to Share Cross-Lingual Knowledge](https://mlsanity.com/paper/2609.19291/) (2026)
- [Do correct answers have stable latent knowledge support?](https://mlsanity.com/paper/2610.15277/) (2026)

## Abstract (arXiv)

The veracity of a factoid is largely independent of the language it is written in. However, language models are inconsistent in their ability to answer the same factual question across languages. This raises questions about how LLMs represent a given fact across languages. We explore multilingual factual knowledge through two aspects: the model's ability to answer a query consistently across languages, and the ability to ''store'' answers in a shared representation for several languages. We propose a methodology to measure the extent of representation sharing across languages by repurposing knowledge editing methods. We examine LLMs with various multilingual configurations using a new multilingual dataset. We reveal that high consistency does not necessarily imply shared representation, particularly for languages with different scripts. Moreover, we find that script similarity is a dominant factor in representation sharing. Finally, we observe that if LLMs could fully share knowledge across languages, their accuracy in their best-performing language could benefit an increase of up to 150% on average. These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.