mlsanityData to 14 Oct 2026

Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages

arXiv 2509.15260 ↗First posted 18 Sep 2025cs.CLcited by 2

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

The advancement of Large Language Models (LLMs) has transformed natural language processing; however, their safety mechanisms remain under-explored in low-resource, multilingual settings. Here, we aim to bridge this gap. In particular, we introduce textsfSGToxicGuard, a novel dataset and evaluation framework for benchmarking LLM safety in Singapore's diverse linguistic context, including Singlish, Chinese, Malay, and Tamil. SGToxicGuard adopts a red-teaming approach to systematically probe LLM vulnerabilities in three real-world scenarios: textitconversation, textitquestion-answering, and textitcontent composition. We conduct extensive experiments with state-of-the-art multilingual LLMs, and the results uncover critical gaps in their safety guardrails. By offering actionable insights into cultural sensitivity and toxicity mitigation, we lay the foundation for safer and more inclusive AI systems in linguistically diverse environments.footnoteLink to the dataset: https://github.com/Social-AI-Studio/SGToxicGuard. textcolorredDisclaimer: This paper contains sensitive content that may be disturbing to some readers.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 26 of its 55 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

26 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming6 Apr

Other references · 25

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2022Red Teaming Language Models with Language Modelscited by 1,026
  5. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  6. 2019How multilingual is Multilingual BERT?cited by 645
  7. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  8. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  9. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  10. 2016Crafting Adversarial Input Sequences for Recurrent Neural Networkscited by 103
  11. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 89
  12. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
13 more
  1. 2022Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Modelscited by 51
  2. 2020Deep Learning Models for Multilingual Hate Speech Detectioncited by 49
  3. 2023Facilitating Fine-grained Detection of Chinese Toxic Language: Hierarchical Taxonomy, Resources, and Benchmarkscited by 46
  4. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 36
  5. 2024ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbationscited by 31
  6. 2021DeepHate: Hate Speech Detection via Multi-Faceted Text Representationscited by 29
  7. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  8. 2021Dataset for Identification of Homophobia and Transophobia in Multilingual YouTube Commentscited by 20
  9. 2021AngryBERT: Joint Learning Target and Emotion for Hate Speech Detectioncited by 19
  10. 2024Arabic Dataset for LLM Safeguard Evaluationcited by 17
  11. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisionscited by 14
  12. 2024SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singaporecited by 10
  13. 2025LionGuard 2: Building Lightweight, Data-Efficient & Localised Multilingual Content Moderatorscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contextscited by 1
  2. 2026Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global Southcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
  2. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
  3. 2025Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps
  4. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?
  5. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding
  6. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages
  7. 2023All Languages Matter: On the Multilingual Safety of Large Language Models
  8. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts