# F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering

arXiv: https://arxiv.org/abs/2010.06283 · HTML: https://mlsanity.com/paper/2010.06283/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 11 of 11)

- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanation](https://mlsanity.com/paper/2201.05088/) (2022)
- [Reasoning over Hierarchical Question Decomposition Tree for Explainable Question Answering](https://mlsanity.com/paper/2305.15056/) (2023)
- [InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations](https://mlsanity.com/paper/2310.05592/) (2023)
- [Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions](https://mlsanity.com/paper/2507.13773/) (2025)
- [What's in your Head? Emergent Behaviour in Multi-Task Transformer Models](https://mlsanity.com/paper/2104.06129/) (2021)
- [Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings](https://mlsanity.com/paper/2109.07833/) (2021)
- [Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering](https://mlsanity.com/paper/2212.10696/) (2022)
- [Question answering systems for health professionals at the point of care -- a systematic review](https://mlsanity.com/paper/2402.01700/) (2024)
- [Challenges in Explanation Quality Evaluation](https://mlsanity.com/paper/2210.07126/) (2022)
- [Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in Large Language Models](https://mlsanity.com/paper/2508.02045/) (2025)

## Abstract (arXiv)

Explainable question answering systems predict an answer together with an explanation showing why the answer has been selected. The goal is to enable users to assess the correctness of the system and understand its reasoning process. However, we show that current models and evaluation settings have shortcomings regarding the coupling of answer and explanation which might cause serious issues in user experience. As a remedy, we propose a hierarchical model and a new regularization term to strengthen the answer-explanation coupling as well as two evaluation scores to quantify the coupling. We conduct experiments on the HOTPOTQA benchmark data set and perform a user study. The user study shows that our models increase the ability of the users to judge the correctness of the system and that scores like F1 are not enough to estimate the usefulness of a model in a practical setting with human users. Our scores are better aligned with user experience, making them promising candidates for model selection.