# HateXScore: A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations

arXiv: https://arxiv.org/abs/2601.13547 · HTML: https://mlsanity.com/paper/2601.13547/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media](https://mlsanity.com/paper/1910.12574/) (2019)
- [Explaining Recurrent Neural Network Predictions in Sentiment Analysis](https://mlsanity.com/paper/1706.07206/) (2017)
- [Probing LLMs for hate speech detection: strengths and vulnerabilities](https://mlsanity.com/paper/2310.12860/) (2023)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [Facilitating Fine-grained Detection of Chinese Toxic Language: Hierarchical Taxonomy, Resources, and Benchmarks](https://mlsanity.com/paper/2305.04446/) (2023)
- [Evaluating GPT-3 Generated Explanations for Hateful Content Moderation](https://mlsanity.com/paper/2305.17680/) (2023)
- [Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement](https://mlsanity.com/paper/2502.06207/) (2025)
- [SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singapore](https://mlsanity.com/paper/2405.01842/) (2024)
- [Explainability and Hate Speech: Structured Explanations Make Social Media Moderators Faster](https://mlsanity.com/paper/2406.04106/) (2024)
- [Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study](https://mlsanity.com/paper/2505.06149/) (2025)
- [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024)
- [Efficient Hate Speech Detection: Evaluating 38 Models from Traditional Methods to Transformers](https://mlsanity.com/paper/2509.14266/) (2025)
- [A Survey on Automatic Online Hate Speech Detection in Low-Resource Languages](https://mlsanity.com/paper/2411.19017/) (2024)

## Abstract (arXiv)

Hateful speech detection is a key component of content moderation, yet current evaluation frameworks rarely assess why a text is deemed hateful. We introduce textsfHateXScore, a four-component metric suite designed to evaluate the reasoning quality of model explanations. It assesses (i) conclusion explicitness, (ii) faithfulness and causal grounding of quoted spans, (iii) protected group identification (policy-configurable), and (iv) logical consistency among these elements. Evaluated on six diverse hate speech datasets, textsfHateXScore is intended as a diagnostic complement to reveal interpretability failures and annotation inconsistencies that are invisible to standard metrics like Accuracy or F1. Moreover, human evaluation shows strong agreement with textsfHateXScore, validating it as a practical tool for trustworthy and transparent moderation. textcolorredDisclaimer: This paper contains sensitive content that may be disturbing to some readers.