# We Need to Talk About Classification Evaluation Metrics in NLP

arXiv: https://arxiv.org/abs/2401.03831 · HTML: https://mlsanity.com/paper/2401.03831/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 23 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Builds on (most cited first, 5 of 5)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Controlling Formality in Low-Resource NMT with Domain Adaptation and Re-Ranking: SLT-CDT-UoS at IWSLT2022](https://mlsanity.com/paper/2205.05990/) (2022)
- [A computationally and cognitively plausible model of supervised and unsupervised learning](https://mlsanity.com/paper/2010.14618/) (2020)

## Built on by (most cited first, 3 of 3)

- [Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning](https://mlsanity.com/paper/2510.12807/) (2025)
- [Language of Persuasion and Misrepresentation in Business Communication: A Textual Detection Approach](https://mlsanity.com/paper/2508.09935/) (2025)
- [TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action](https://mlsanity.com/paper/2509.15098/) (2025)

## Abstract (arXiv)

In Natural Language Processing (NLP) classification tasks such as topic categorisation and sentiment analysis, model generalizability is generally measured with standard metrics such as Accuracy, F-Measure, or AUC-ROC. The diversity of metrics, and the arbitrariness of their application suggest that there is no agreement within NLP on a single best metric to use. This lack suggests there has not been sufficient examination of the underlying heuristics which each metric encodes. To address this we compare several standard classification metrics with more 'exotic' metrics and demonstrate that a random-guess normalised Informedness metric is a parsimonious baseline for task performance. To show how important the choice of metric is, we perform extensive experiments on a wide range of NLP tasks including a synthetic scenario, natural language understanding, question answering and machine translation. Across these tasks we use a superset of metrics to rank models and find that Informedness best captures the ideal model characteristics. Finally, we release a Python implementation of Informedness following the SciKitLearn classifier format.