Used, Mentioned, or Condemned? A Controlled Contrast-Set Diagnostic for the Use-Mention Distinction in Code-Mixed Hinglish Misogyny Detection
Not on the map. Why?
Abstract (arXiv)
Lexicon-driven misogyny detectors cannot, by construction, distinguish a slur used against a woman from the same slur mentioned in counter-speech ("don't call her that") -- yet exactly this distinction governs whether moderation protects or silences the people discussing abuse. We study this problem in code-mixed Hinglish and make three contributions. First, we diagnose two evaluation artifacts on a publicly available redacted corpus: category-encoding anonymization placeholders leak the label (a no-learning rule scores 1.000), and even after they are neutralized misogynistic and benign comments occupy lexically disjoint registers, so bag-of-words reaches macro-F1 approximately 1.00 under random cross-validation but collapses under template-disjoint evaluation. Second, we release Hinglish-MGY-Diag, a deterministic generator and a 416-item / 163-minimal-pair contrast-set diagnostic across five linguistically motivated categories in which slur presence and gendered register are decorrelated from the label by construction. Third, we introduce a strict pair-consistency metric that credits a model only when both members of a minimal pair are correctly labelled. Five from-scratch classical baselines evaluated under construction-disjoint five-fold cross-validation reveal that the strongest model reaches 0.93 accuracy on the cleanest use-mention subset but only 0.82 consistency -- it still mislabels roughly one counter-speech pair in five. A frontier LLM used as an author-model ceiling attains 1.000 on all metrics, doubling as independent label validation and confirming the benchmark is a capability gradient rather than an adversarial wall. We release all code, data, the generator, and an arms-length LLM harness for reproducing every number.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 24 references that are in the map. First posted on arXiv 6 Sep 2026.
Builds on
8 of 24 references.
References · 8
- 2018Datasheets for Datasetscited by 1,371
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
- 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
- 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
- 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
- 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
- 2022HateCheckHIn: Evaluating Hindi Hate Speech Detection Modelscited by 15
Built on by
Not cited yet.