# Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities

arXiv: https://arxiv.org/abs/2505.15722 · HTML: https://mlsanity.com/paper/2505.15722/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [On the Privacy Risks of Algorithmic Fairness](https://mlsanity.com/paper/2011.03731/) (2020)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023)
- [Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?](https://mlsanity.com/paper/1909.01638/) (2019)
- [Memorization Through the Lens of Curvature of Loss Function Around Samples](https://mlsanity.com/paper/2307.05831/) (2023)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Base](https://mlsanity.com/paper/2409.18472/) (2024)
- [Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations](https://mlsanity.com/paper/2401.05792/) (2024)
- [On the Privacy Effect of Data Enhancement via the Lens of Memorization](https://mlsanity.com/paper/2208.08270/) (2022)
- [Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models](https://mlsanity.com/paper/2410.04727/) (2024)
- [Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks](https://mlsanity.com/paper/2101.10649/) (2021)
- [ROME: Memorization Insights from Text, Logits and Representation](https://mlsanity.com/paper/2403.00510/) (2024)
- [A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Base](https://mlsanity.com/paper/2405.11125/) (2024)
- [Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis](https://mlsanity.com/paper/2410.13237/) (2024)
- [Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyond](https://mlsanity.com/paper/2503.07501/) (2025)

## Built on by (most cited first, 3 of 3)

- [Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects](https://mlsanity.com/paper/2603.02333/) (2026)
- [Beyond Frequency: The Role of Redundancy in Large Language Model Memorization](https://mlsanity.com/paper/2506.12321/) (2025)
- [Cross-Lingual Transferability of Training Data Extraction Attacks to Recover Memorized PII](https://mlsanity.com/paper/2610.06093/) (2026)

## Abstract (arXiv)

We present the first comprehensive study of Memorization in Multilingual Large Language Models (MLLMs), analyzing 95 languages using models across diverse model scales, architectures, and memorization definitions. As MLLMs are increasingly deployed, understanding their memorization behavior has become critical. Yet prior work has focused primarily on monolingual models, leaving multilingual memorization underexplored, despite the inherently long-tailed nature of training corpora. We find that the prevailing assumption, that memorization is highly correlated with training data availability, fails to fully explain memorization patterns in MLLMs. We hypothesize that the conventional focus on monolingual settings, effectively treating languages in isolation, may obscure the true patterns of memorization. To address this, we propose a novel graph-based correlation metric that incorporates language similarity to analyze cross-lingual memorization. Our analysis reveals that among similar languages, those with fewer training tokens tend to exhibit higher memorization, a trend that only emerges when cross-lingual relationships are explicitly modeled. These findings underscore the importance of a textitlanguage-aware perspective in evaluating and mitigating memorization vulnerabilities in MLLMs. This also constitutes empirical evidence that language similarity both explains Memorization in MLLMs and underpins Cross-lingual Transferability, with broad implications for multilingual NLP.