# Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

arXiv: https://arxiv.org/abs/2609.40286 · HTML: https://mlsanity.com/paper/2609.40286/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 31)

- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://mlsanity.com/paper/2412.03304/) (2024)
- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics](https://mlsanity.com/paper/2506.12618/) (2025)
- [DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models](https://mlsanity.com/paper/2310.20138/) (2023)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models](https://mlsanity.com/paper/2410.17509/) (2024)
- [AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages](https://mlsanity.com/paper/2311.09828/) (2023)
- [Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models](https://mlsanity.com/paper/2410.11718/) (2024)
- [LUME: LLM Unlearning with Multitask Evaluations](https://mlsanity.com/paper/2502.15097/) (2025)
- [Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2406.12354/) (2024)
- [Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge](https://mlsanity.com/paper/2403.05189/) (2024)
- [Learn and Unlearn: Addressing Misinformation in Multilingual LLMs](https://mlsanity.com/paper/2406.13748/) (2024)
- [Does Localization Inform Unlearning? A Rigorous Examination of Local Parameter Attribution for Knowledge Unlearning in Language Models](https://mlsanity.com/paper/2505.16252/) (2025)
- [Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs](https://mlsanity.com/paper/2601.05641/) (2026)
- [Uncovering the Potential Risks in Unlearning: Danger of English-only Unlearning in Multilingual LLMs](https://mlsanity.com/paper/2510.23949/) (2025)
- [Evaluating Cross-Lingual Unlearning in Multilingual Language Models](https://mlsanity.com/paper/2601.06675/) (2026)
- [Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation](https://mlsanity.com/paper/2605.14404/) (2026)
- [Layer-Targeted Multilingual Knowledge Erasure in Large Language Models](https://mlsanity.com/paper/2602.22562/) (2026)
- [Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility](https://mlsanity.com/paper/2606.03291/) (2026)
- [FAME: Fictional Actors for Multilingual Erasure](https://mlsanity.com/paper/2512.15235/) (2025)

## Abstract (arXiv)

Unlearning a fact in one language does not guarantee its removal in others as changing the query or even the requested answer language can reopen seemingly forgotten knowledge -- a cross-lingual loophole. The most straightforward solution to this challenge -- unlearning in all languages -- is neither scalable nor desirable as it amplifies damage to unrelated model capabilities. We introduce the task of language budgeted multilingual unlearning where the goal is to select a subset of languages that maximizes cross-lingual erasure. To study this task we introduce the Cross-Lingual Unlearning Tensor, an unlearning benchmark that spans 174 language--script pairs and 25 atomic paraphrase types to examine when forgetting generalizes across linguistic expressions of the same knowledge. We further propose COVER, which selects source languages to maximize predicted COVERage of languages receiving no forget supervision, enabling unlearning on a language budget. Surprisingly, we find naively selecting strong individual sources does not reliably compose into strong source sets motivating our development of COVER. At deployment COVER only requires benign calibration data and access to the frozen model. Across three model families and two disjoint forget sets, COVER reduces mean held-out residual access by 7.8--27.3% relative to uniform source selection. We find these gains extend beyond synthetic benchmarks to real news documents in low-resource language settings using human translated data from the Low Resource Languages for Emergent Incidents (LORELEI) corpus.