# All Languages Matter: On the Multilingual Safety of Large Language Models

arXiv: https://arxiv.org/abs/2310.00905 · HTML: https://mlsanity.com/paper/2310.00905/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Challenges in Detoxifying Language Models](https://mlsanity.com/paper/2109.07445/) (2021)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Safety Assessment of Chinese Large Language Models](https://mlsanity.com/paper/2304.10436/) (2023)
- [Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models](https://mlsanity.com/paper/2202.04173/) (2022)
- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [BiasAsker: Measuring the Bias in Conversational AI System](https://mlsanity.com/paper/2305.12434/) (2023)
- [SafeText: A Benchmark for Exploring Physical Safety in Language Models](https://mlsanity.com/paper/2210.10045/) (2022)

## Built on by (most cited first, 30 of 53)

- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm](https://mlsanity.com/paper/2406.18682/) (2024)
- [JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models](https://mlsanity.com/paper/2407.01599/) (2024)
- [Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training](https://mlsanity.com/paper/2407.09121/) (2024)
- [PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages](https://mlsanity.com/paper/2504.04377/) (2025)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Models](https://mlsanity.com/paper/2401.10647/) (2024)
- [PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models](https://mlsanity.com/paper/2405.09373/) (2024)
- [Mapping the Ethics of Generative AI: A Comprehensive Scoping Review](https://mlsanity.com/paper/2402.08323/) (2024)
- [Enhancing LLM Safety Through a Theoretical Minimax Game Lens](https://mlsanity.com/paper/2502.05163/) (2025)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learning](https://mlsanity.com/paper/2410.10801/) (2024)
- [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://mlsanity.com/paper/2401.00763/) (2024)
- [CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2406.10311/) (2024)
- [Agentic Abstention: Do Agents Know When to Stop Instead of Act?](https://mlsanity.com/paper/2606.28733/) (2026)
- [Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Search](https://mlsanity.com/paper/2602.22983/) (2026)
- [Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs](https://mlsanity.com/paper/2412.16974/) (2024)
- [K2-Think: A Parameter-Efficient Reasoning System](https://mlsanity.com/paper/2509.07604/) (2025)
- [MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety](https://mlsanity.com/paper/2504.15241/) (2025)
- [Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment](https://mlsanity.com/paper/2602.16660/) (2026)
- [POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization](https://mlsanity.com/paper/2410.12999/) (2024)
- [LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies](https://mlsanity.com/paper/2412.15035/) (2024)
- [On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning](https://mlsanity.com/paper/2408.14398/) (2024)
- [Learn to Disguise: Avoid Refusal Responses in LLM's Defense via a Multi-agent Attacker-Disguiser Game](https://mlsanity.com/paper/2404.02532/) (2024)
- [JailBench: A Comprehensive Chinese Security Assessment Benchmark for Large Language Models](https://mlsanity.com/paper/2502.18935/) (2025)
- [Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data](https://mlsanity.com/paper/2506.00469/) (2025)

## Abstract (arXiv)

Safety lies at the core of developing and deploying large language models (LLMs). However, previous safety benchmarks only concern the safety in one language, e.g. the majority language in the pretraining data such as English. In this work, we build the first multilingual safety benchmark for LLMs, XSafety, in response to the global deployment of LLMs in practice. XSafety covers 14 kinds of commonly used safety issues across 10 languages that span several language families. We utilize XSafety to empirically study the multilingual safety for 4 widely-used LLMs, including both close-API and open-source models. Experimental results show that all LLMs produce significantly more unsafe responses for non-English queries than English ones, indicating the necessity of developing safety alignment for non-English languages. In addition, we propose several simple and effective prompting methods to improve the multilingual safety of ChatGPT by evoking safety knowledge and improving cross-lingual generalization of safety alignment. Our prompting method can significantly reduce the ratio of unsafe responses from 19.1% to 9.7% for non-English queries. We release our data at https://github.com/Jarviswang94/Multilingual_safety_benchmark.