mlsanityData to 7 Oct 2026

Used, Mentioned, or Condemned? A Controlled Contrast-Set Diagnostic for the Use-Mention Distinction in Code-Mixed Hinglish Misogyny Detection

arXiv 2609.22261 ↗First posted 6 Sep 2026cs.CL cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Lexicon-driven misogyny detectors cannot, by construction, distinguish a slur used against a woman from the same slur mentioned in counter-speech ("don't call her that") -- yet exactly this distinction governs whether moderation protects or silences the people discussing abuse. We study this problem in code-mixed Hinglish and make three contributions. First, we diagnose two evaluation artifacts on a publicly available redacted corpus: category-encoding anonymization placeholders leak the label (a no-learning rule scores 1.000), and even after they are neutralized misogynistic and benign comments occupy lexically disjoint registers, so bag-of-words reaches macro-F1 approximately 1.00 under random cross-validation but collapses under template-disjoint evaluation. Second, we release Hinglish-MGY-Diag, a deterministic generator and a 416-item / 163-minimal-pair contrast-set diagnostic across five linguistically motivated categories in which slur presence and gendered register are decorrelated from the label by construction. Third, we introduce a strict pair-consistency metric that credits a model only when both members of a minimal pair are correctly labelled. Five from-scratch classical baselines evaluated under construction-disjoint five-fold cross-validation reveal that the strongest model reaches 0.93 accuracy on the cleanest use-mention subset but only 0.82 consistency -- it still mislabels roughly one counter-speech pair in five. A frontier LLM used as an author-model ceiling attains 1.000 on all metrics, doubling as independent label validation and confirming the benchmark is a capability gradient rather than an adversarial wall. We release all code, data, the generator, and an arms-length LLM harness for reproducing every number.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 24 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

8 of 24 references.

References · 8

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  4. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  5. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  6. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  7. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  8. 2022HateCheckHIn: Evaluating Hindi Hate Speech Detection Modelscited by 15

Built on by

Not cited yet.