# The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning

arXiv: https://arxiv.org/abs/2606.28843 · HTML: https://mlsanity.com/paper/2606.28843/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 18 of 18)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal](https://mlsanity.com/paper/2406.14598/) (2024)
- [LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B](https://mlsanity.com/paper/2310.20624/) (2023)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.16534/) (2025)
- [Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks](https://mlsanity.com/paper/2410.18210/) (2024)
- [Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency](https://mlsanity.com/paper/2506.17209/) (2025)
- [The effect of fine-tuning on language model toxicity](https://mlsanity.com/paper/2410.15821/) (2024)
- [Refusal Behavior in Large Language Models: A Nonlinear Perspective](https://mlsanity.com/paper/2501.08145/) (2025)
- [Understanding and Preserving Safety in Fine-Tuned LLMs](https://mlsanity.com/paper/2601.10141/) (2026)
- [Curvature-Aware Safety Restoration In LLMs Fine-Tuning](https://mlsanity.com/paper/2511.18039/) (2025)

## Abstract (arXiv)

Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability comes with a cost: it can increase a model's tendency to respond to unsafe adversarial prompts, even when fine-tuning with non-adversarial data. We present the first comprehensive empirical study of this phenomenon in multilingual settings by fine-tuning Llama-3.2, Qwen3, and Gemma-3 models using benign data translated across nine languages. We find that safety outcomes are highly sensitive to both the choice of fine-tuning language and the evaluation language, with adversarial compliance rates increasing four-fold in some settings. Multilingual safety drift is decoupled from general capability metrics, and occurs heterogeneously across languages and models. Fine-tuning in non-English languages often induces smaller internal representational drifts than English, but these shifts lead models to default to either exaggerated compliance or refusal. As such, assessing fine-tuning impacts solely in English provides inadequate assurance for deployment. To facilitate further research into these cross-lingual safety blind spots, we release the Multilingual-Benign-Tune dataset and the SORRY-Bench-Multilingual evaluation suite.