# XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

arXiv: https://arxiv.org/abs/2605.05662 · HTML: https://mlsanity.com/paper/2605.05662/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 29 of 29)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm](https://mlsanity.com/paper/2406.18682/) (2024)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance](https://mlsanity.com/paper/2507.22448/) (2025)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models](https://mlsanity.com/paper/2404.12464/) (2024)
- [The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It](https://mlsanity.com/paper/2505.24119/) (2025)
- [PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data](https://mlsanity.com/paper/2406.15053/) (2024)
- [LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs](https://mlsanity.com/paper/2407.03963/) (2024)
- [Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs](https://mlsanity.com/paper/2410.03730/) (2024)
- [Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs](https://mlsanity.com/paper/2502.12982/) (2025)
- [LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2508.12733/) (2025)
- [RakutenAI-7B: Extending Large Language Models for Japanese](https://mlsanity.com/paper/2403.15484/) (2024)
- [K2-Think: A Parameter-Efficient Reasoning System](https://mlsanity.com/paper/2509.07604/) (2025)
- [The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation](https://mlsanity.com/paper/2503.12294/) (2025)
- [What Has Been Lost with Synthetic Evaluation?](https://mlsanity.com/paper/2505.22830/) (2025)
- [Gaperon: A Peppered English-French Generative Language Model Suite](https://mlsanity.com/paper/2510.25771/) (2025)
- [JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks](https://mlsanity.com/paper/2603.01291/) (2026)
- [K-EXAONE Technical Report](https://mlsanity.com/paper/2601.01739/) (2026)
- [RigoChat 2: an adapted language model to Spanish using a bounded dataset and reduced hardware](https://mlsanity.com/paper/2503.08188/) (2025)

## Built on by (most cited first, 2 of 2)

- [EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures](https://mlsanity.com/paper/2606.30219/) (2026)
- [Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability](https://mlsanity.com/paper/2607.06196/) (2026)

## Abstract (arXiv)

Current LLM safety benchmarks are predominantly English-centric and often rely on translation, failing to capture country-specific harms. Moreover, they rarely evaluate a model's ability to detect culturally embedded sensitivities as distinct from universal harms. We introduce XL-SafetyBench, a suite of 5,500 test cases across 10 country-language pairs, comprising a Jailbreak Benchmark of country-grounded adversarial prompts and a Cultural Benchmark where local sensitivities are embedded within innocuous requests. Each item is constructed via a multi-stage pipeline that combines LLM-assisted discovery, automated validation gates, and dual independent native-speaker annotators per country. We report Attack Success Rate (ASR) alongside Neutral-Safe Rate (NSR) to separate principled refusal from non-engagement, and introduce the Cultural Sensitivity Rate (CSR) for recognizing embedded sensitivities. Evaluating 10 frontier and 27 local LLMs reveals two key findings. First, strong jailbreak robustness does not always come with strong cultural awareness, showing why the two should be evaluated separately. Second, local models exhibit a strong ASR-NSR trade-off (r = -0.81), suggesting that low attack success in several local models reflects non-engagement rather than safe refusal. XL-SafetyBench enables more nuanced, cross-cultural safety evaluation in the multilingual era.