The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning
Not on the map. Why?
Abstract (arXiv)
Fine-tuning a large language model is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability comes with a cost: it can increase a model's tendency to respond to unsafe adversarial prompts, even when fine-tuning with non-adversarial data. We present the first comprehensive empirical study of this phenomenon in multilingual settings by fine-tuning Llama-3.2, Qwen3, and Gemma-3 models using benign data translated across nine languages. We find that safety outcomes are highly sensitive to both the choice of fine-tuning language and the evaluation language, with adversarial compliance rates increasing four-fold in some settings. Multilingual safety drift is decoupled from general capability metrics, and occurs heterogeneously across languages and models. Fine-tuning in non-English languages often induces smaller internal representational drifts than English, but these shifts lead models to default to either exaggerated compliance or refusal. As such, assessing fine-tuning impacts solely in English provides inadequate assurance for deployment. To facilitate further research into these cross-lingual safety blind spots, we release the Multilingual-Benign-Tune dataset and the SORRY-Bench-Multilingual evaluation suite.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 27 Jun 2026.
Builds on
18 of 27 references.
References · 18
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
- 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
- 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
- 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
- 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
- 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
- 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
- 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
- 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
6 more
- 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
- 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 27
- 2024The effect of fine-tuning on language model toxicitycited by 16
- 2025Refusal Behavior in Large Language Models: A Nonlinear Perspectivecited by 13
- 2026Understanding and Preserving Safety in Fine-Tuned LLMscited by 7
- 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuningcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
- 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
- 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning
- 2026Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
- 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
- 2026Lost in Translation? A Comparative Study on the Cross-Lingual Transfer of Composite Harms
- 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
- 2025Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards