mlsanityData to 14 Oct 2026

Benchmarking LLM Guardrails in Handling Multilingual Toxicity

arXiv 2410.22153 ↗First posted 29 Oct 2024cs.CLcited by 14

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

With the ubiquity of Large Language Models (LLMs), guardrails have become crucial to detect and defend against toxic content. However, with the increasing pervasiveness of LLMs in multilingual scenarios, their effectiveness in handling multilingual toxic inputs remains unclear. In this work, we introduce a comprehensive multilingual test suite, spanning seven datasets and over ten languages, to benchmark the performance of state-of-the-art guardrails. We also investigates the resilience of guardrails against recent jailbreaking techniques, and assess the impact of in-context safety policies and language resource availability on guardrails' performance. Our findings show that existing guardrails are still ineffective at handling multilingual toxicity and lack robustness against jailbreaking prompts. This work aims to identify the limitations of guardrails and to build a more reliable and trustworthy LLMs in multilingual scenarios.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 18 of its 31 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

18 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning8 Jul

Other references · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  8. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  9. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  10. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  11. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  12. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 68
5 more
  1. 2022Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Modelscited by 51
  2. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?cited by 36
  3. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  4. 2020Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXELcited by 23
  5. 2019Towards Robust Toxic Content Classificationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 2.

14 papers.

Sort:

By area:

  1. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
  2. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  3. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 19
  4. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  5. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  6. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 9
  7. 2024LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistenciescited by 7
  8. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 7
  9. 2025Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attackscited by 5
  10. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 3
  11. 2025CREST: Universal Safety Guardrails Through Cluster-Guided Cross-Lingual Transfercited by 2
  12. 2026SEA-Guard: Culturally Grounded Multilingual Safeguard for Southeast Asiacited by 2
2 more
  1. 2025"Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selectioncited by 1
  2. 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Models
  2. 2025SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systems
  3. 2026Why Do Safety Guardrails Degrade Across Languages?
  4. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
  5. 2026Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
  6. 2025Auto-Tuning Safety Guardrails for Black-Box Large Language Models
  7. 2025Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models
  8. 2023Multilingual Jailbreak Challenges in Large Language Models