# Being Right for Whose Right Reasons?

arXiv: https://arxiv.org/abs/2306.00639 · HTML: https://mlsanity.com/paper/2306.00639/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 67 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 30 of 32)

- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [A Clarification of the Nuances in the Fairness Metrics Landscape](https://mlsanity.com/paper/2106.00467/) (2021)
- [Explaining Recurrent Neural Network Predictions in Sentiment Analysis](https://mlsanity.com/paper/1706.07206/) (2017)
- [Fairness risk measures](https://mlsanity.com/paper/1901.08665/) (2019)
- [Meaning without reference in large language models](https://mlsanity.com/paper/2208.02957/) (2022)
- [DynaSent: A Dynamic Benchmark for Sentiment Analysis](https://mlsanity.com/paper/2012.15349/) (2020)
- [Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation](https://mlsanity.com/paper/1710.06169/) (2017)
- [The Pragmatic Turn in Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2002.09595/) (2020)
- [What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play](https://mlsanity.com/paper/1810.09648/) (2018)
- [FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing](https://mlsanity.com/paper/2203.07228/) (2022)
- [GLocalX -- From Local to Global Explanations of Black Box AI Models](https://mlsanity.com/paper/2101.07685/) (2021)
- [Your fairness may vary: Pretrained language model fairness in toxic text classification](https://mlsanity.com/paper/2108.01250/) (2021)
- [Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold](https://mlsanity.com/paper/2206.09755/) (2022)
- [Can Model Compression Improve NLP Fairness](https://mlsanity.com/paper/2201.08542/) (2022)
- [Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models](https://mlsanity.com/paper/2206.03945/) (2022)
- [A Fine-grained Interpretability Evaluation Benchmark for Neural NLP](https://mlsanity.com/paper/2205.11097/) (2022)
- [On the Interaction of Belief Bias and Explanations](https://mlsanity.com/paper/2106.15355/) (2021)
- [Machine Reading, Fast and Slow: When Do Models "Understand" Language?](https://mlsanity.com/paper/2209.07430/) (2022)

## Built on by (most cited first, 12 of 12)

- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [Rather a Nurse than a Physician -- Contrastive Explanations under Investigation](https://mlsanity.com/paper/2310.11906/) (2023)
- [On the Interplay between Fairness and Explainability](https://mlsanity.com/paper/2310.16607/) (2023)
- [Persona Prompting as a Lens on LLM Social Reasoning](https://mlsanity.com/paper/2601.20757/) (2026)
- [Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Models](https://mlsanity.com/paper/2505.19621/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [A Systematic Comparison between Extractive Self-Explanations and Human Rationales in Text Classification](https://mlsanity.com/paper/2410.03296/) (2024)
- [Learning from Sufficient Rationales: Analysing the Relationship Between Explanation Faithfulness and Token-level Regularisation Strategies](https://mlsanity.com/paper/2511.16353/) (2025)
- [Evaluating Webcam-based Gaze Data as an Alternative for Human Rationale Annotations](https://mlsanity.com/paper/2402.19133/) (2024)
- [Automated Trustworthiness Testing for Machine Learning Classifiers](https://mlsanity.com/paper/2406.05251/) (2024)
- [Beyond Technocratic XAI: The Who, What & How in Explanation Design](https://mlsanity.com/paper/2508.09231/) (2025)

## Abstract (arXiv)

Explainability methods are used to benchmark the extent to which model predictions align with human rationales i.e., are 'right for the right reasons'. Previous work has failed to acknowledge, however, that what counts as a rationale is sometimes subjective. This paper presents what we think is a first of its kind, a collection of human rationale annotations augmented with the annotators demographic information. We cover three datasets spanning sentiment analysis and common-sense reasoning, and six demographic groups (balanced across age and ethnicity). Such data enables us to ask both what demographics our predictions align with and whose reasoning patterns our models' rationales align with. We find systematic inter-group annotator disagreement and show how 16 Transformer-based models align better with rationales provided by certain demographic groups: We find that models are biased towards aligning best with older and/or white annotators. We zoom in on the effects of model size and model distillation, finding -- contrary to our expectations -- negative correlations between model size and rationale agreement as well as no evidence that either model size or model distillation improves fairness.