# Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?

arXiv: https://arxiv.org/abs/2311.03788 · HTML: https://mlsanity.com/paper/2311.03788/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 7 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [Editable Neural Networks](https://mlsanity.com/paper/2004.00345/) (2020)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [Inducing Language-Agnostic Multilingual Representations](https://mlsanity.com/paper/2008.09112/) (2020)
- [Consistency Regularization for Cross-Lingual Fine-Tuning](https://mlsanity.com/paper/2106.08226/) (2021)
- [Enhancing Cross-lingual Transfer by Manifold Mixup](https://mlsanity.com/paper/2205.04182/) (2022)
- [Language Anisotropic Cross-Lingual Model Editing](https://mlsanity.com/paper/2205.12677/) (2022)

## Built on by (most cited first, 19 of 19)

- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?](https://mlsanity.com/paper/2402.18120/) (2024)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [Concept Space Alignment in Multilingual LLMs](https://mlsanity.com/paper/2410.01079/) (2024)
- [Enhancing Non-English Capabilities of English-Centric Large Language Models through Deep Supervision Fine-Tuning](https://mlsanity.com/paper/2503.01275/) (2025)
- [FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data](https://mlsanity.com/paper/2408.06273/) (2024)
- [Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models](https://mlsanity.com/paper/2406.16135/) (2024)
- [Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations](https://mlsanity.com/paper/2504.13816/) (2025)
- [Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis](https://mlsanity.com/paper/2505.24593/) (2025)
- [Language Fusion for Parameter-Efficient Cross-lingual Transfer](https://mlsanity.com/paper/2501.06892/) (2025)
- [Pruning Multilingual Large Language Models for Multilingual Inference](https://mlsanity.com/paper/2409.16911/) (2024)
- [Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops](https://mlsanity.com/paper/2608.28860/) (2026)
- [Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation](https://mlsanity.com/paper/2505.21106/) (2025)
- [Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages](https://mlsanity.com/paper/2509.17493/) (2025)
- [Code-Switching Reveals Language Anchoring in Multilingual LLMs](https://mlsanity.com/paper/2606.19668/) (2026)
- [Assessing Agentic Large Language Models in Multilingual National Bias](https://mlsanity.com/paper/2502.17945/) (2025)
- [Probing Factual Knowledge Transfer with Training Data Interventions](https://mlsanity.com/paper/2609.01341/) (2026)

## Abstract (arXiv)

Multilingual pretrained language models serve as repositories of multilingual factual knowledge. Nevertheless, a substantial performance gap of factual knowledge probing exists between high-resource languages and low-resource languages, suggesting limited implicit factual knowledge transfer across languages in multilingual pretrained language models. This paper investigates the feasibility of explicitly transferring relatively rich factual knowledge from English to non-English languages. To accomplish this, we propose two parameter-free Language Representation Projection modules (LRP2). The first module converts non-English representations into English-like equivalents, while the second module reverts English-like representations back into representations of the corresponding non-English language. Experimental results on the mLAMA dataset demonstrate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transferability across diverse non-English languages. We further investigate the working mechanism of LRP2 from the perspectives of representation space and cross-lingual knowledge neuron.