# Semantic Pivots Enable Cross-Lingual Transfer in Large Language Models

arXiv: https://arxiv.org/abs/2505.16385 · HTML: https://mlsanity.com/paper/2505.16385/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://mlsanity.com/paper/2401.17377/) (2024)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [What's In My Big Data?](https://mlsanity.com/paper/2310.20707/) (2023)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [PolyLM: An Open Source Polyglot Large Language Model](https://mlsanity.com/paper/2307.06018/) (2023)
- [BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages](https://mlsanity.com/paper/2305.18098/) (2023)
- [PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning](https://mlsanity.com/paper/2311.08711/) (2023)
- [Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models](https://mlsanity.com/paper/2205.11758/) (2022)

## Built on by (most cited first, 1 of 1)

- [Rethinking Pivot Programming Languages in Code Language Models](https://mlsanity.com/paper/2609.22988/) (2026)

## Abstract (arXiv)

Large language models (LLMs) demonstrate remarkable ability in cross-lingual tasks. Understanding how LLMs acquire this ability is crucial for their interpretability. To quantify the cross-lingual ability of LLMs accurately, we propose a Word-Level Cross-Lingual Translation Task. To find how LLMs learn cross-lingual ability, we trace the outputs of LLMs' intermediate layers in the word translation task. We identify and distinguish two distinct behaviors in the forward pass of LLMs: co-occurrence behavior and semantic pivot behavior. We attribute LLMs' two distinct behaviors to the co-occurrence frequency of words and find the semantic pivot from the pre-training dataset. Finally, to apply our findings to improve the cross-lingual ability of LLMs, we reconstruct a semantic pivot-aware pre-training dataset using documents with a high proportion of semantic pivots. Our experiments validate the effectiveness of our approach in enhancing cross-lingual ability. Our research contributes insights into the interpretability of LLMs and offers a method for improving LLMs' cross-lingual ability.