# Evaluating Classification Systems Against Soft Labels with Fuzzy Precision and Recall

arXiv: https://arxiv.org/abs/2309.13938 · HTML: https://mlsanity.com/paper/2309.13938/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 6 of 6)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Threshold Independent Evaluation of Sound Event Detection Scores](https://mlsanity.com/paper/2201.13148/) (2022)
- [Beyond Hard Labels: Investigating data label distributions](https://mlsanity.com/paper/2207.06224/) (2022)
- [Training sound event detection with soft labels from crowdsourced annotations](https://mlsanity.com/paper/2302.14572/) (2023)

## Built on by (most cited first, 3 of 3)

- [Training and Evaluating with Human Label Variation: An Empirical Study](https://mlsanity.com/paper/2502.01891/) (2025)
- [Prominence-Aware Artifact Detection and Dataset for Image Super-Resolution](https://mlsanity.com/paper/2510.16752/) (2025)
- [A Novel Method to Evaluate Models on Unreliable, Noisy and Inconsistent Labels: Adaptive Resolution Label Aggregation (ARLA)](https://mlsanity.com/paper/2607.11214/) (2026)

## Abstract (arXiv)

Classification systems are normally trained by minimizing the cross-entropy between system outputs and reference labels, which makes the Kullback-Leibler divergence a natural choice for measuring how closely the system can follow the data. Precision and recall provide another perspective for measuring the performance of a classification system. Non-binary references can arise from various sources, and it is often beneficial to use the soft labels for training instead of the binarized data. However, the existing definitions for precision and recall require binary reference labels, and binarizing the data can cause erroneous interpretations. We present a novel method to calculate precision, recall and F-score without quantizing the data. The proposed metrics extend the well established metrics as the definitions coincide when used with binary labels. To understand the behavior of the metrics we show simple example cases and an evaluation of different sound event detection models trained on real data with soft labels.