When Surface Form Changes Moderation Decisions: A Paired Study of Code-Mixed Workflow Instability
Not on the map. Why?
Abstract (arXiv)
Hate moderation is often evaluated as classification on clean English inputs, but deployed systems must route content to actions such as ALLOW, FLAG, or REVIEW. We study how this workflow changes under code-mixed inputs using a paired evaluation setting where the same underlying content is expressed as clean English and Tamil-English code-mix. Under thresholds tuned on clean English development data, code-mixed inputs produce substantial action instability, with a paired clean- to-code-mix decision flip rate of 0.265. The main workflow effects are increased review burden and increased false-flagging of non-hateful content: review rate rises from 0.138 to 0.297 and non-hate false-flag rate rises from 0.069 to 0.104. Tamil-only inputs show stronger degradation overall, suggesting a broader language-coverage limitation rather than the same code-mixed instability pattern. A simple disagreement-based deferral rule reduces automatic errors on stressed inputs, but only by increasing review load. These results show that workflow-level evaluation reveals moderation failures that standard classification summaries can miss.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 21 references that are in the map. First posted on arXiv 4 Jun 2026.
Builds on
14 of 21 references.
References · 14
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
- 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
- 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 270
- 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
- 2021Classification with abstention but without disparitiescited by 28
- 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 25
- 2021Uncertainty-Aware Reliable Text Classificationcited by 23
- 2020Multidimensional Uncertainty-Aware Evidential Neural Networkscited by 16
- 2025HateBench: Benchmarking Hate Speech Detectors on LLM-Generated Content and Hate Campaignscited by 14
- 2025CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languagescited by 10
- 2026Can Large Language Models Understand, Reason About, and Generate Code-Switched Text?cited by 5
Built on by
Not cited yet.