mlsanityData to 14 Oct 2026

The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It

arXiv 2505.24119 ↗First posted 30 May 2025cs.CLcited by 45

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

This paper presents a comprehensive analysis of the linguistic diversity of LLM safety research, highlighting the English-centric nature of the field. Through a systematic review of nearly 300 publications from 2020--2024 across major NLP conferences and workshops at *ACL, we identify a significant and growing language gap in LLM safety research, with even high-resource non-English languages receiving minimal attention. We further observe that non-English languages are rarely studied as a standalone language and that English safety research exhibits poor language documentation practice. To motivate future research into multilingual safety, we make several recommendations based on our survey, and we then pose three concrete future directions on safety evaluation, training data generation, and crosslingual safety generalization. Based on our survey and proposed directions, the field can develop more robust, inclusive AI safety practices for diverse global populations.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 88 of its 141 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

88 of 141 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimization22 May
  2. 2024Large Language Model Safety: A Holistic Survey23 Dec
  3. 2024SafeWorld: Geo-Diverse Safety Alignment9 Dec
  4. 2024Benchmarking LLM Guardrails in Handling Multilingual Toxicity29 Oct
  5. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks23 Oct
  6. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm26 Jun
  7. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunch20 Jun
  8. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety8 Apr

Other references · 80

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  7. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  8. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  9. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  10. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  11. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  12. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
38 more
  1. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  3. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  4. 2023Cultural Bias and Cultural Alignment of Large Language Modelscited by 149
  5. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  6. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  7. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
  8. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  9. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 127
  10. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  11. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  12. 2021Gender Bias in Machine Translationcited by 117
  13. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  14. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  15. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  16. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  17. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 72
  18. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  19. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 64
  20. 2025Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Modelscited by 62
  21. 2025Command A: An Enterprise-Ready Large Language Modelcited by 59
  22. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  23. 2024Open Problems in Technical AI Governancecited by 49
  24. 2023Specific versus General Principles for Constitutional AIcited by 47
  25. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  26. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  27. 2024Evaluating Cultural and Social Awareness of LLM Web Agentscited by 45
  28. 2024AI auditing: The Broken Bus on the Road to AI Accountabilitycited by 43
  29. 2024A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontierscited by 40
  30. 2023Fake Alignment: Are LLMs Really Aligned Well?cited by 39
  31. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 38
  32. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  33. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  34. 2025Aya Vision: Advancing the Frontier of Multilingual Multimodalitycited by 33
  35. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  36. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  37. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  38. 2024Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Toolingcited by 29

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 38.

45 papers.

Sort:

By area:

  1. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 15
  2. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 12
  3. 2026Safe Language Generation in the Limitcited by 9
  4. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  5. 2025Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safetycited by 4
  6. 2026LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safetycited by 3
  7. 2026Multilingual Safety Alignment via Self-Distillationcited by 3
  8. 2025Mina: A Multilingual LLM-Powered Legal Assistant Agent for Bangladesh for Empowering Access to Justicecited by 2
  9. 2026XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivitycited by 2
  10. 2026Low-Resource Safety Failures Are Action Failures, Not Representation Failurescited by 2
  11. 2026Multilingual Refusal Alignment for Safer Large Language Modelscited by 2
  12. 2026The Culture Funnel: You Can't Align What isn't in the Datacited by 2
33 more
  1. 2025Policy-Grounded Safety Evaluation of 20 Large Language Modelscited by 1
  2. 2025The Problem with Safety Classification is not just the Modelscited by 1
  3. 2025BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culturecited by 1
  4. 2026JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Modelscited by 1
  5. 2026Alignment Backfire: Language-Dependent Reversal of Safety Interventions Across 16 Languages in LLM Multi-Agent Systemscited by 1
  6. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  7. 2026Multilingual jailbreaking of LLMs using low-resource languagescited by 1
  8. 2026AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albaniancited by 1
  9. 2026The Illusion of Cross-Lingual Safety in Low-Resource Languagescited by 1
  10. 2025Mirror, Mirror on the Wall -- Which is the Best Model of Them All?cited by 0
  11. 2026Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarincited by 0
  12. 2026BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Storiescited by 0
  13. 2026Cross-Lingual Jailbreak Detection via Semantic Codebookscited by 0
  14. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0
  15. 2026ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safetycited by 0
  16. 2026SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Modelscited by 0
  17. 2026Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinesecited by 0
  18. 2026RiskNet: A large-scale dataset of AI risk incidents from news with alignment and multi-dimensional annotationscited by 0
  19. 2026Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysiscited by 0
  20. 2026Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Modelscited by 0
  21. 2026Jailbreaking for the Average Jane: Choosing Optimal Jailbreaks via Bandit Algorithms for Automatically Enhanced Queriescited by 0
  22. 2026World Wide Models: Literary Tools for Cultural AIcited by 0
  23. 2026LLM Scheming Inversely Scales with Pretraining Language Coveragecited by 0
  24. 2026Same violence, different answer: how AI responds to coercive control against women across languagescited by 0
  25. 2026Aligned in Form, Not in Meaning: The Comprehension - Containment Decoupling of LLM Safety in Low-Resource Bangla Derogatory Speechcited by 0
  26. 2026Where Privacy Risk Lives in English-Source Multilingual RAG: A Stage-Decomposed Audit Across Five Query Languagescited by 0
  27. 2026LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Reviewcited by 0
  28. 2026Exploratory As-Analyzed No-Detection of Culturally-Marked Predicate-Triggered PII Amplification in a Synthetic-English RAG Probe: A Predicate-Resource-Confounded Auditcited by 0
  29. 2026BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Modelscited by 0
  30. 2026Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languagescited by 0
  31. 2026When Safety Speaks a Language: A Mechanistic Analysis of Safety-Language Identity Entanglement in LLMscited by 0
  32. 2026ArGuard Shared Task: Harmful Content Detection in Arabic Memes and LLM Promptscited by 0
  33. 2026Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages
  2. 2026IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
  3. 2026TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts
  4. 2026Sch"utzen: Evaluating LLM Safety in Bulgarian and German Contexts
  5. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models
  6. 2025Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
  8. 2023All Languages Matter: On the Multilingual Safety of Large Language Models