# A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice

arXiv: https://arxiv.org/abs/2404.16958 · HTML: https://mlsanity.com/paper/2404.16958/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 54 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 21 of 21)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)
- [Metrics for Multi-Class Classification: an Overview](https://mlsanity.com/paper/2008.05756/) (2020)
- [Parsing Argumentation Structures in Persuasive Essays](https://mlsanity.com/paper/1604.07370/) (2016)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Macro F1 and Macro F1](https://mlsanity.com/paper/1911.03347/) (2019)
- [SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images](https://mlsanity.com/paper/2105.09284/) (2021)
- [Evaluation Measures for Hierarchical Classification: a unified view and novel approaches](https://mlsanity.com/paper/1306.6802/) (2013)
- [Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators](https://mlsanity.com/paper/1904.13015/) (2019)
- [Interpretable Multi-dataset Evaluation for Named Entity Recognition](https://mlsanity.com/paper/2011.06854/) (2020)
- [Practical Text Classification With Large Pre-Trained Language Models](https://mlsanity.com/paper/1812.01207/) (2018)
- [Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels](https://mlsanity.com/paper/2311.05265/) (2023)
- [BB_twtr at SemEval-2017 Task 4: Twitter Sentiment Analysis with CNNs and LSTMs](https://mlsanity.com/paper/1704.06125/) (2017)
- [What the F-measure doesn't measure: Features, Flaws, Fallacies and Fixes](https://mlsanity.com/paper/1503.06410/) (2015)
- [An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results](https://mlsanity.com/paper/2006.01245/) (2020)
- [Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference](https://mlsanity.com/paper/2010.03760/) (2020)
- [Epicurus at SemEval-2023 Task 4: Improving Prediction of Human Values behind Arguments by Leveraging Their Definitions](https://mlsanity.com/paper/2302.13925/) (2023)
- [Senti17 at SemEval-2017 Task 4: Ten Convolutional Neural Network Voters for Tweet Polarity Classification](https://mlsanity.com/paper/1705.02023/) (2017)

## Built on by (most cited first, 26 of 26)

- [Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents](https://mlsanity.com/paper/2604.16706/) (2026)
- [PARAPHRASUS : A Comprehensive Benchmark for Evaluating Paraphrase Detection Models](https://mlsanity.com/paper/2409.12060/) (2024)
- [Sentence Smith: Controllable Edits for Evaluating Text Embeddings](https://mlsanity.com/paper/2502.14734/) (2025)
- [Lessons from the trenches on evaluating machine-learning systems in materials science](https://mlsanity.com/paper/2503.10837/) (2025)
- [CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluation](https://mlsanity.com/paper/2511.18889/) (2025)
- [Benchmarking graph construction by large language models for coherence-driven inference](https://mlsanity.com/paper/2502.13953/) (2025)
- [Investigating the Impact of Rational Dilated Wavelet Transform on Motor Imagery EEG Decoding with Deep Learning Models](https://mlsanity.com/paper/2510.09242/) (2025)
- [CLEF HIPE-2026: Evaluating Accurate and Efficient Person-Place Relation Extraction from Multilingual Historical Texts](https://mlsanity.com/paper/2602.17663/) (2026)
- [Overview of HIPE-2026: Person-Place Relation Extraction from Multilingual Historical Texts](https://mlsanity.com/paper/2606.25935/) (2026)
- [Pioneering Precision in Lumbar Spine MRI Segmentation with Advanced Deep Learning and Data Enhancement](https://mlsanity.com/paper/2409.06018/) (2024)
- [The Representation of Meaningful Precision, and Accuracy](https://mlsanity.com/paper/2410.14721/) (2024)
- [Multidimensional classification of posts for online course discussion forum curation](https://mlsanity.com/paper/2508.10008/) (2025)
- [Engineering Spatial and Molecular Features from Cellular Niches to Inform Predictions of Inflammatory Bowel Disease](https://mlsanity.com/paper/2509.09923/) (2025)
- [TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action](https://mlsanity.com/paper/2509.15098/) (2025)
- [RatioWaveNet: A Learnable RDWT Front-End for Robust and Interpretable EEG Motor-Imagery Classification](https://mlsanity.com/paper/2510.21841/) (2025)
- [LxCIM: a new rank-based binary classifier performance metric invariant to local exchange of classes](https://mlsanity.com/paper/2512.10053/) (2025)
- [Transforming Vehicle Diagnostics: A Multimodal Approach to Error Patterns Prediction](https://mlsanity.com/paper/2602.01109/) (2026)
- [Multilevel neural networks with dual-stage feature fusion for human activity recognition](https://mlsanity.com/paper/2604.16577/) (2026)
- [Radiomic Feature Selection Using Gradient Loss of Deep Neural Network for Lung Cancer Stage Detection](https://mlsanity.com/paper/2606.04453/) (2026)
- [Benchmarking Large Language Models for Safety Data Extraction](https://mlsanity.com/paper/2606.11204/) (2026)
- [Robustness of neural networks to random noise perturbations of their inputs](https://mlsanity.com/paper/2606.31581/) (2026)
- [Lightweight Person-Place Relation Extraction from Historical Newspapers with Dependency Graphs and Proximity Features](https://mlsanity.com/paper/2607.19718/) (2026)
- [BayesSeg: A Bayesian Optimization Framework for State Segmentation of Electricity Consumption Time Series](https://mlsanity.com/paper/2608.00513/) (2026)
- [Disentangling Homophily and Rarity: Explaining Failure in Graph Neural Networks](https://mlsanity.com/paper/2608.14823/) (2026)
- [From Noise to Signal: Improving Security Log Anomaly Detection Using LLMs with Endpoint-Specific Logs](https://mlsanity.com/paper/2608.19938/) (2026)
- [Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing](https://mlsanity.com/paper/2609.28026/) (2026)

## Abstract (arXiv)

Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without arguments, and blurry terminology invites misconceptions. For instance, many works use so-called 'macro' metrics to rank systems (e.g., 'macro F1') but do not clearly specify what they would expect from such a `macro' metric. This is problematic, since picking a metric can affect research findings, and thus any clarity in the process should be maximized. Starting from the intuitive concepts of bias and prevalence, we perform an analysis of common evaluation metrics. The analysis helps us understand the metrics' underlying properties, and how they align with expectations as found expressed in papers. Then we reflect on the practical situation in the field, and survey evaluation practice in recent shared tasks. We find that metric selection is often not supported with convincing arguments, an issue that can make a system ranking seem arbitrary. Our work aims at providing overview and guidance for more informed and transparent metric selection, fostering meaningful evaluation.