'Ghaib in Translation' aka Unseen Harm: Measuring Cross-Script Safety Inconsistency with 'Missed-in-Urdu' Scores in LLM Hate Speech Detection
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Urdu, the world's tenth most spoken language with 246 million speakers, remains almost entirely absent from mainstream LLM safety evaluation and nine years of WOAH proceedings. To investigate whether this absence has measurable consequences for content moderation reliability, five large language models, GPT-4o, Claude Sonnet 4.5, Gemini 2.5 Flash, Qwen-2.5, and Llama-3.1, were tested across six datasets spanning Nastaliq Urdu, Roman Urdu, English, and code-switched Urdu-English. Across the five Urdu-script datasets, label instability between original-script and English-translation classification ranged from 15.9% (Gemini 2.5 Flash) to 31.6% (Qwen-2.5), with a 'Missed-in-Urdu' rate, content flagged as harmful in English translation but passed as normal in the original script, ranging from 2.4% to 9.9% (median 4.3%). A complete enumeration of all 205 papers across nine ALW/WOAH editions via the ACL Anthology API confirms zero dedicated Urdu papers across the entire period. Results indicate that current LLMs provide uneven safety assurance across Urdu's script varieties, with smaller open-weight models showing substantially higher instability and missed-harm rates than frontier closed models.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 9 of its 38 references that are in the map. First posted on arXiv 25 Aug 2026.
Builds on
9 of 38 references.
Closest ancestors · 1
Same direction, newest first
Other references · 8
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
- 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
- 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
- 2019Multilingual and Multi-Aspect Hate Speech Analysiscited by 133
- 2024Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settingscited by 14
- 2021Hate Speech Detection in Roman Urducited by 8
- 2025Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Studycited by 7
Built on by
Not cited yet.