# Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

arXiv: https://arxiv.org/abs/2411.10954 · HTML: https://mlsanity.com/paper/2411.10954/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Builds on (most cited first, 15 of 15)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023)
- [Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://mlsanity.com/paper/2212.08011/) (2022)
- [Probing LLMs for hate speech detection: strengths and vulnerabilities](https://mlsanity.com/paper/2310.12860/) (2023)
- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [SD-QA: Spoken Dialectal Question Answering for the Real World](https://mlsanity.com/paper/2109.12072/) (2021)
- [Evaluation of African American Language Bias in Natural Language Generation](https://mlsanity.com/paper/2305.14291/) (2023)
- [Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets](https://mlsanity.com/paper/2401.16313/) (2024)
- [Automatic Dialect Adaptation in Finnish and its Effect on Perceived Creativity](https://mlsanity.com/paper/2009.02685/) (2020)
- [Lemmatization of Historical Old Literary Finnish Texts in Modern Orthography](https://mlsanity.com/paper/2107.03266/) (2021)
- [Normalization of Different Swedish Dialects Spoken in Finland](https://mlsanity.com/paper/2012.05318/) (2020)

## Built on by (most cited first, 12 of 12)

- [Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge](https://mlsanity.com/paper/2601.13649/) (2026)
- [LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories](https://mlsanity.com/paper/2605.31381/) (2026)
- [Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages](https://mlsanity.com/paper/2607.02235/) (2026)
- [Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation](https://mlsanity.com/paper/2608.00794/) (2026)
- [A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic)](https://mlsanity.com/paper/2505.04640/) (2025)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)
- [textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language](https://mlsanity.com/paper/2603.28213/) (2026)
- [Aligned in Form, Not in Meaning: The Comprehension - Containment Decoupling of LLM Safety in Low-Resource Bangla Derogatory Speech](https://mlsanity.com/paper/2608.02941/) (2026)
- [Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelines](https://mlsanity.com/paper/2608.29589/) (2026)
- [Probing the Structure and Dynamics of LLM Value Expression through Value Conflicts](https://mlsanity.com/paper/2609.07296/) (2026)
- [Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery](https://mlsanity.com/paper/2610.00164/) (2026)
- [Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and English](https://mlsanity.com/paper/2610.18343/) (2026)

## Abstract (arXiv)

There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators ("LLM-as-a-judge") is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluated three LLMs on their ability to assess toxicity across multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive in handling both multilingual and dialectal variations. However, if we have to rank the consistency, the weakest area is LLM-human agreement, followed by dialectal consistency. Code repository: urlhttps://github.com/ffaisal93/dialecttoxicityllmjudge