# Language-Specific Gaps in AI Safety Training Datasets

arXiv: https://arxiv.org/abs/2608.13695 · HTML: https://mlsanity.com/paper/2608.13695/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm](https://mlsanity.com/paper/2406.18682/) (2024)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applications](https://mlsanity.com/paper/2508.01710/) (2025)
- [Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments](https://mlsanity.com/paper/2203.11764/) (2022)
- [AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages](https://mlsanity.com/paper/2501.08284/) (2025)
- [LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies](https://mlsanity.com/paper/2412.15035/) (2024)
- [ToxBuster: In-game Chat Toxicity Buster with BERT](https://mlsanity.com/paper/2305.12542/) (2023)
- [Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource-Efficient Toxicity Detection](https://mlsanity.com/paper/2506.06347/) (2025)
- [UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages](https://mlsanity.com/paper/2601.12696/) (2026)
- [Detection and Analysis of Offensive Online Content in Hausa Language](https://mlsanity.com/paper/2311.10541/) (2023)
- [TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages](https://mlsanity.com/paper/2606.01322/) (2026)
- [LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages](https://mlsanity.com/paper/2603.19273/) (2026)

## Abstract (arXiv)

Large language model providers routinely cite multilingual safety benchmarks spanning a dozen or more languages as evidence that their models are safe for non-English-speaking users. We show that these collection-level coverage claims frequently do not survive inspection at the level of an individual language. Auditing 21 resources across 25 language slices, of which 20 count as datasets under our counting rules, spanning three languages chosen to represent low- (Hausa), mid- (Swahili), and high-resource (French) tiers, we find that gaps in provenance, annotation reliability, access, harm-taxonomy coverage, and data reuse recur in patterns that partially, but not fully, track resource level. Using a controlled within-pipeline comparison, we show a Hausa-language slice falling below its own paper's translation-quality acceptance threshold while the same pipeline's Swahili output clears the same bar comfortably; this is evidence that these gaps are measurable and addressable, not inherent. We further show that self-harm and sexual-content categories have no native-language coverage in either African-language tier we studied, a total rather than gradated gap that a purely resource-level account does not predict. We connect these findings to a documented, persistent asymmetry in multilingual jailbreak robustness (single-turn attacks largely mitigated, multi-turn attacks still effective), arguing that this asymmetry is structurally consistent with where our audit finds training and evaluation data thinnest. We contribute a reusable slice-level audit methodology, a cross-tier empirical comparison, and concrete recommendations for dataset creators, model providers, and venues aiming to make ``multilingual coverage'' claims verifiable rather than merely stated. Dataset: https://huggingface.co/datasets/ChialukaOnuoha/safety-slice-audit