# Why Do Safety Guardrails Degrade Across Languages?

arXiv: https://arxiv.org/abs/2605.17173 · HTML: https://mlsanity.com/paper/2605.17173/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 16 May 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 22 of 22)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [What is in Your Safe Data? Identifying Benign Data that Breaks Safety](https://mlsanity.com/paper/2404.01099/) (2024)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts](https://mlsanity.com/paper/2401.13136/) (2024)
- [All Languages Matter: On the Multilingual Safety of Large Language Models](https://mlsanity.com/paper/2310.00905/) (2023)
- [The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions](https://mlsanity.com/paper/2502.09674/) (2025)
- [Refusal Direction is Universal Across Safety-Aligned Languages](https://mlsanity.com/paper/2505.17306/) (2025)
- [Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2508.06709/) (2025)
- [Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts](https://mlsanity.com/paper/2503.09347/) (2025)
- [LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2508.12733/) (2025)
- [Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming](https://mlsanity.com/paper/2601.18076/) (2026)
- [Concept Space Alignment in Multilingual LLMs](https://mlsanity.com/paper/2410.01079/) (2024)
- [Fantastic Bugs and Where to Find Them in AI Benchmarks](https://mlsanity.com/paper/2511.16842/) (2025)
- [LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies](https://mlsanity.com/paper/2412.15035/) (2024)

## Abstract (arXiv)

Large language models exhibit safety degradation in non-English languages. Standard evaluation relies on Jailbreak Success Rate (JSR), which confounds several safety-driving factors into one, obscuring the specific cause(s) of safety failure. We introduce a latent variable model, a Multi-Group Item Response Theory (IRT) framework, that decouples language-agnostic safety robustness (θ), intrinsic prompt hardness (β), global language processing difficulty (γ), and a prompt-specific cross-lingual safety gap (τ). Using the MultiJail dataset, we evaluate the safety robustness of 61 model configurations across 5 closed-model families and 10 languages of varying resource, aggregating a dataset of 1.9 million responses. Exploratory Factor Analysis shows safety is primarily unidimensional: models refuse different harm types mainly through a shared mechanism. Contrary to the expected trend that safety degrades largely in low-resource languages, 22 model configurations are more vulnerable in English than in low-resource languages. Low-resource languages produce more uncertain responses (high entropy) than high-resource languages. Also, high-τ prompts cluster in physical harm categories like Theft and Weapons and lower-resource languages, trends validated through cross-dataset generalization. While global translation quality shows low correlation with τ, severe mistranslations drive high-bias outliers, as validated by native speakers. Cultural and conceptual grounding mismatches may also contribute to τ. In predictive validation, the IRT framework achieves AUC = 0.940, and unlike rate baselines stays predictive when a whole language is held out (0.875). Our framework reveals concept-language vulnerabilities that aggregate metrics obscure, enabling fairer cross-lingual safety evaluation and targeted improvements in dataset construction.