mlsanityData to 14 Oct 2026

MPO: Multilingual Safety Alignment via Reward Gap Optimization

arXiv 2505.16869 ↗First posted 22 May 2025cs.CLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have become increasingly central to AI applications worldwide, necessitating robust multilingual safety alignment to ensure secure deployment across diverse linguistic contexts. Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data. To address these limitations, we introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages. MPO directly minimizes the reward gap difference between the dominant language and target languages, effectively transferring safety capabilities while preserving the original strengths of the dominant language. Extensive experiments on three LLMs, LLaMA-3.1, Gemma-2 and Qwen2.5, validate MPO's efficacy in multilingual safety alignment without degrading general multilingual utility.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 85 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

64 of 85 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context3 Apr
  2. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture10 Jul
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  4. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts23 Jan

Other references · 60

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
38 more
  1. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2023LIMA: Less Is More for Alignmentcited by 1,100
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  10. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  11. 2025LIMO: Less is More for Reasoningcited by 498
  12. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  13. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  14. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  15. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  16. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  17. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  18. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  19. 2024A Roadmap to Pluralistic Alignmentcited by 184
  20. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  21. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  22. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  23. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  24. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
  25. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 96
  26. 2023Do Multilingual Language Models Think Better in English?cited by 88
  27. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 81
  28. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  29. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 69
  30. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  31. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 50
  32. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  33. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 35
  34. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  35. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  36. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  37. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  38. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  2. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 21
  3. 2026Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearningcited by 10
  4. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  5. 2026Multilingual Safety Alignment Via Sparse Weight Editingcited by 6
  6. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?cited by 5
  7. 2025SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfacescited by 4
  8. 2026LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safetycited by 3
  9. 2026Multilingual Safety Alignment via Self-Distillationcited by 3
  10. 2026Low-Resource Safety Failures Are Action Failures, Not Representation Failurescited by 2
  11. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  12. 2026Learning to Route Languages for Multilingual Policy Optimizationcited by 1
6 more
  1. 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languagescited by 1
  2. 2025SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Culturescited by 0
  3. 2026LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Reviewcited by 0
  4. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimizationcited by 0
  5. 2026When Safety Speaks a Language: A Mechanistic Analysis of Safety-Language Identity Entanglement in LLMscited by 0
  6. 2026ACTR: Aligning Thoughts and Responses for Multilingual Safety in Reasoning LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BSO: Safety Alignment Is Density Ratio Matching
  2. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling
  3. 2025Safe at the Margins: A General Approach to Safety Alignment in Low-Resource English Languages -- A Singlish Case Study
  4. 2026Multilingual Refusal Alignment for Safer Large Language Models
  5. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
  6. 2026Improving Safety Alignment via Balanced Direct Preference Optimization
  7. 2026Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
  8. 2026Cat-DPO: Category-Adaptive Safety Alignment