mlsanityData to 14 Oct 2026

All Languages Matter: On the Multilingual Safety of Large Language Models

arXiv 2310.00905 ↗First posted 2 Oct 2023cs.CL cs.AIcited by 53

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Safety lies at the core of developing and deploying large language models (LLMs). However, previous safety benchmarks only concern the safety in one language, e.g. the majority language in the pretraining data such as English. In this work, we build the first multilingual safety benchmark for LLMs, XSafety, in response to the global deployment of LLMs in practice. XSafety covers 14 kinds of commonly used safety issues across 10 languages that span several language families. We utilize XSafety to empirically study the multilingual safety for 4 widely-used LLMs, including both close-API and open-source models. Experimental results show that all LLMs produce significantly more unsafe responses for non-English queries than English ones, indicating the necessity of developing safety alignment for non-English languages. In addition, we propose several simple and effective prompting methods to improve the multilingual safety of ChatGPT by evoking safety knowledge and improving cross-lingual generalization of safety alignment. Our prompting method can significantly reduce the ratio of unsafe responses from 19.1% to 9.7% for non-English queries. We release our data at https://github.com/Jarviswang94/Multilingual_safety_benchmark.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 28 of its 48 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

28 of 48 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2022Red Teaming Language Models with Language Modelscited by 1,026
  10. 2021Ethical and social risks of harm from Language Modelscited by 966
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
16 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  4. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  5. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  6. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  7. 2022Prompting GPT-3 To Be Reliablecited by 260
  8. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  9. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  10. 2021Challenges in Detoxifying Language Modelscited by 169
  11. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  12. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  13. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 68
  14. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  15. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 53
  16. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 20; 2025: 13; 2026: 18.

53 papers.

Sort:

By area:

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  2. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  4. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  5. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  6. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  7. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  8. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
  9. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 58
  10. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  11. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20
  12. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 20
38 more
  1. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  2. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 19
  3. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  4. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  5. 2024New Job, New Gender? Measuring the Social Bias in Image Generation Modelscited by 16
  6. 2024CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Modelscited by 15
  7. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 15
  8. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13
  9. 2024Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMscited by 11
  10. 2025K2-Think: A Parameter-Efficient Reasoning Systemcited by 11
  11. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  12. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  13. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
  14. 2024LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistenciescited by 7
  15. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  16. 2024Learn to Disguise: Avoid Refusal Responses in LLM's Defense via a Multi-agent Attacker-Disguiser Gamecited by 5
  17. 2025JailBench: A Comprehensive Chinese Security Assessment Benchmark for Large Language Modelscited by 5
  18. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  19. 2025Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safetycited by 4
  20. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Modelscited by 3
  21. 2026Trust The Typicalcited by 3
  22. 2026CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generationcited by 3
  23. 2026Multilingual Safety Alignment via Self-Distillationcited by 3
  24. 2024Towards Safe Multilingual Frontier AIcited by 2
  25. 2024Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)cited by 2
  26. 2025Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreakcited by 2
  27. 2025How Catastrophic is Your LLM? Certifying Risk in Conversationcited by 2
  28. 2026Layer-Targeted Multilingual Knowledge Erasure in Large Language Modelscited by 2
  29. 2024Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generationcited by 1
  30. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluationcited by 1
  31. 2025Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrievalcited by 0
  32. 2026ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contextscited by 0
  33. 2026Why Do Safety Guardrails Degrade Across Languages?cited by 0
  34. 2026Opir: Efficient Multi-Task Safety Classification for Toxicity, Jailbreaks, Hate Speech, and Harmful Contentcited by 0
  35. 2026Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysiscited by 0
  36. 2026Safety That Does Not Transfer: Cross-Lingual Clinical Correctness Drift in Deployable Medical Language Modelscited by 0
  37. 2026Safety Cost of Steering Vectors Is Separable and Reduciblecited by 0
  38. 2026Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot Seecited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models
  2. 2026LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review
  3. 2026IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
  4. 2026Multilingual Safety Signals Are Multi-Layered: Filtering Safety-Degrading Data for Safer LLMs
  5. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture
  6. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
  7. 2025SafeLawBench: Towards Safe Alignment of Large Language Models
  8. 2025CREST: Universal Safety Guardrails Through Cluster-Guided Cross-Lingual Transfer