mlsanityData to 14 Oct 2026

Preference Tuning For Toxicity Mitigation Generalizes Across Languages

arXiv 2406.16235 ↗First posted 23 Jun 2024cs.CL cs.AI cs.CRcited by 31

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Detoxifying multilingual Large Language Models (LLMs) has become crucial due to their increasing global use. In this work, we explore zero-shot cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike previous studies that show limited cross-lingual generalization for other safety tasks, we demonstrate that Direct Preference Optimization (DPO) training with only English data can significantly reduce toxicity in multilingual open-ended generations. For example, the probability of mGPT-1.3B generating toxic continuations drops from 46.8% to 3.9% across 17 different languages after training. Our results also extend to other multilingual LLMs, such as BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal intervention and activation analysis, we identified the dual multilinguality property of MLP layers in LLMs, which explains the cross-lingual generalization of DPO. Finally, we show that bilingual sentence retrieval can predict the cross-lingual transferability of DPO preference tuning.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 67 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

45 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  2. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity3 Jan

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
31 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  4. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  6. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  7. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  11. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  12. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  13. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  14. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  15. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  16. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  17. 2020A Distributional Approach to Controlled Text Generationcited by 119
  18. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  19. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  20. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  21. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 47
  22. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?cited by 36
  23. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  24. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  25. 2023Concrete Problems in AI Safety, Revisitedcited by 22
  26. 2024mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?cited by 21
  27. 2024From One to Many: Expanding the Scope of Toxicity Mitigation in Language Modelscited by 18
  28. 2023Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxificationcited by 15
  29. 2024MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languagescited by 14
  30. 2024STAR: SocioTechnical Approach to Red Teaming Language Modelscited by 13
  31. 2024"I Searched for a Religious Song in Amharic and Got Sexual Content Instead": Investigating Online Harm in Low-Resourced Languages on YouTubecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 13; 2026: 6.

31 papers.

Sort:

By area:

  1. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  2. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  3. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  4. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  5. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  6. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  7. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methodscited by 26
  8. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  9. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  10. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  11. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  12. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
19 more
  1. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  2. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 9
  3. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  4. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigationcited by 5
  5. 2025Aligned Probing: Relating Toxic Behavior and Model Internalscited by 5
  6. 2025Breaking mBad! Supervised Fine-tuning for Cross-Lingual Detoxificationcited by 3
  7. 2025ToxiFrench: Benchmarking and Enhancing Language Models via CoT Fine-Tuning for French Toxicity Detectioncited by 3
  8. 2024Reverse-Engineering the Readercited by 2
  9. 2025Long-context Non-factoid Question Answering in Indic Languagescited by 2
  10. 2025Something Just Like TRuST : Toxicity Recognition of Span and Targetcited by 2
  11. 2025Social Bias in Multilingual Language Models: A Surveycited by 2
  12. 2025Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Modelscited by 1
  13. 2026BabelSteering: Multilingual Safety Alignment via English Steering Vectorscited by 1
  14. 2025Multilingual Collaborative Defense for Large Language Modelscited by 0
  15. 2026Cyberbullying Governance on Social Media: A Unified Framework from Content Identification to Interventioncited by 0
  16. 2026Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Modelscited by 0
  17. 2026A Survey of Toxicity Detection and Mitigation Strategies for Multilingual Language Modelscited by 0
  18. 2026Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retrainingcited by 0
  19. 2026Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Detoxifying LLMs via Representation Erasure-Based Preference Optimization
  2. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context
  3. 2025Detoxification of Large Language Models through Output-layer Fusion with a Calibration Model
  4. 2024Large Language Models can be Strong Self-Detoxifiers
  5. 2024Realistic Evaluation of Toxicity in Large Language Models
  6. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
  7. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
  8. 2026The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning