# Disagreeing Rationales: Rethinking Classification and Explainability Evaluation in Hate Speech Detection

arXiv: https://arxiv.org/abs/2605.31563 · HTML: https://mlsanity.com/paper/2605.31563/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 20 of 20)

- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Evaluating and Aggregating Feature-based Model Explanations](https://mlsanity.com/paper/2005.00631/) (2020)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Do Human Rationales Improve Machine Explanations?](https://mlsanity.com/paper/1905.13714/) (2019)
- ["Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?](https://mlsanity.com/paper/2406.17600/) (2024)
- [Ecologically Valid Explanations for Label Variation in NLI](https://mlsanity.com/paper/2310.13850/) (2023)
- [A survey on improving NLP models with human explanations](https://mlsanity.com/paper/2204.08892/) (2022)
- [A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI](https://mlsanity.com/paper/2412.13942/) (2024)
- [Training and Evaluating with Human Label Variation: An Empirical Study](https://mlsanity.com/paper/2502.01891/) (2025)
- [Unifying VXAI: A Systematic Review and Framework for the Evaluation of Explainable AI](https://mlsanity.com/paper/2506.15408/) (2025)
- [Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems](https://mlsanity.com/paper/2506.20209/) (2025)
- [Aligning Attention with Human Rationales for Self-Explaining Hate Speech Detection](https://mlsanity.com/paper/2511.07065/) (2025)
- [EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models](https://mlsanity.com/paper/2505.01238/) (2025)

## Built on by (most cited first, 1 of 1)

- [From Plausible to Actionable: A Position on LLM Self-Explanations](https://mlsanity.com/paper/2607.15957/) (2026)

## Abstract (arXiv)

Human disagreement is ubiquitous and well-known in labeling. However, variation in explanations, captured through token-level human rationales, remains far less explored. At the same time, it is unclear how to best evaluate human labels and rationales -- or even how to best aggregate rationales beyond majority vote -- in light of this variation. Yet, rationales may provide additional insights into the richness of human reasoning, that may differ in style, values and interpretations -- especially in subjective NLP tasks like hate speech detection. In this work, we unify diverse models, training strategies, loss functions, and existing evaluation metrics under a single protocol by systematically re-implementing them across different label and rationale representation spaces. Classification metrics are organized around two key properties -- predictive and distributional -- while explainability metrics through three complementary dimensions: plausibility, faithfulness, and complexity. In this unified supervision framework, we evaluate model behavior across classification and explainability metrics, as well as metric sensitivity to the choice of label (hard and soft) and rationale representation space (hard, intermediate and soft). Results show that both hard and soft metrics favor softer representations, highlighting their effectiveness in capturing variation and the need to rethink evaluation in subjective NLP.