# LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2508.12733 · HTML: https://mlsanity.com/paper/2508.12733/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 5 of its 15 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 5 of 5)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [Rule Based Rewards for Language Model Safety](https://mlsanity.com/paper/2411.01111/) (2024)

## Built on by (most cited first, 16 of 16)

- [JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks](https://mlsanity.com/paper/2603.01291/) (2026)
- [Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks](https://mlsanity.com/paper/2511.22047/) (2025)
- [Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators](https://mlsanity.com/paper/2606.07874/) (2026)
- [XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity](https://mlsanity.com/paper/2605.05662/) (2026)
- [Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B](https://mlsanity.com/paper/2510.08624/) (2025)
- [JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models](https://mlsanity.com/paper/2601.01627/) (2026)
- [Alignment Backfire: Language-Dependent Reversal of Safety Interventions Across 16 Languages in LLM Multi-Agent Systems](https://mlsanity.com/paper/2603.04904/) (2026)
- [BabelSafe: A Policy-Grounded Multilingual Safety Benchmark for LLMs](https://mlsanity.com/paper/2605.00689/) (2026)
- [When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels](https://mlsanity.com/paper/2605.06652/) (2026)
- [On the Robustness of Knowledge Editing for Detoxification](https://mlsanity.com/paper/2602.10504/) (2026)
- [ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety](https://mlsanity.com/paper/2605.14152/) (2026)
- [Why Do Safety Guardrails Degrade Across Languages?](https://mlsanity.com/paper/2605.17173/) (2026)
- [Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathways](https://mlsanity.com/paper/2608.09095/) (2026)
- [Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languages](https://mlsanity.com/paper/2608.22490/) (2026)
- [When Safety Speaks a Language: A Mechanistic Analysis of Safety-Language Identity Entanglement in LLMs](https://mlsanity.com/paper/2608.29936/) (2026)
- [Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models](https://mlsanity.com/paper/2610.15134/) (2026)

## Abstract (arXiv)

The widespread adoption and increasing prominence of large language models (LLMs) in global technologies necessitate a rigorous focus on ensuring their safety across a diverse range of linguistic and cultural contexts. The lack of a comprehensive evaluation and diverse data in existing multilingual safety evaluations for LLMs limits their effectiveness, hindering the development of robust multilingual safety alignment. To address this critical gap, we introduce LinguaSafe, a comprehensive multilingual safety benchmark crafted with meticulous attention to linguistic authenticity. The LinguaSafe dataset comprises 45k entries in 12 languages, ranging from Hungarian to Malay. Curated using a combination of translated, transcreated, and natively-sourced data, our dataset addresses the critical need for multilingual safety evaluations of LLMs, filling the void in the safety evaluation of LLMs across diverse under-represented languages from Hungarian to Malay. LinguaSafe presents a multidimensional and fine-grained evaluation framework, with direct and indirect safety assessments, including further evaluations for oversensitivity. The results of safety and helpfulness evaluations vary significantly across different domains and different languages, even in languages with similar resource levels. Our benchmark provides a comprehensive suite of metrics for in-depth safety evaluation, underscoring the critical importance of thoroughly assessing multilingual safety in LLMs to achieve more balanced safety alignment. Our dataset and code are released to the public to facilitate further research in the field of multilingual LLM safety.