mlsanityData to 14 Oct 2026

Anomaly Detection: How to Artificially Increase your F1-Score with a Biased Evaluation Protocol

arXiv 2106.16020 ↗First posted 30 Jun 2021cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Anomaly detection is a widely explored domain in machine learning. Many models are proposed in the literature, and compared through different metrics measured on various datasets. The most popular metrics used to compare performances are F1-score, AUC and AVPR. In this paper, we show that F1-score and AVPR are highly sensitive to the contamination rate. One consequence is that it is possible to artificially increase their values by modifying the train-test split procedure. This leads to misleading comparisons between algorithms in the literature, especially when the evaluation protocol is not well detailed. Moreover, we show that the F1-score and the AVPR cannot be used to compare performances on different datasets as they do not reflect the intrinsic difficulty of modeling such data. Based on these observations, we claim that F1-score and AVPR should not be used as metrics for anomaly detection. We recommend a generic evaluation procedure for unsupervised anomaly detection, including the use of other metrics such as the AUC, which are more robust to arbitrary choices in the evaluation protocol.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 12 of its 31 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

12 of 31 references.

References · 12

  1. 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
  2. 2018GANomaly: Semi-Supervised Anomaly Detection via Adversarial Trainingcited by 409
  3. 2019OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representationscited by 199
  4. 2020DROCC: Deep Robust One-Class Classificationcited by 121
  5. 2020Old is Gold: Redefining the Adversarially Learned One-Class Classifier Training Paradigmcited by 115
  6. 2016Deep Structured Energy Based Models for Anomaly Detectioncited by 114
  7. 2017Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networkscited by 40
  8. 2019Robust Subspace Recovery Layer for Unsupervised Anomaly Detectioncited by 31
  9. 2020GAN Ensemble for Anomaly Detectioncited by 22
  10. 2019Robust Variational Autoencoders for Outlier Detection and Repair of Mixed-Type Datacited by 16
  11. 2020Memory Augmented Generative Adversarial Networks for Anomaly Detectioncited by 12
  12. 2020Regularized Cycle Consistent Generative Adversarial Network for Anomaly Detectioncited by 6

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 0; 2024: 4.

9 papers.

  1. 2022Estimating the Contamination Factor's Distribution in Unsupervised Anomaly Detectioncited by 14
  2. 2022A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithmscited by 11
  3. 2022Deep-Disaster: Unsupervised Disaster Detection and Localization Using Visual Datacited by 3
  4. 2022Unsupervised Anomaly Detection for Auditing Data and Impact of Categorical Encodingscited by 3
  5. 2024FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Datacited by 3
  6. 2022Automatic Tuberculosis and COVID-19 cough classification using deep learningcited by 2
  7. 2024Benchmarking Anomaly Detection Algorithms: Deep Learning and Beyondcited by 1
  8. 2024Introducing 'Inside' Out of Distributioncited by 1
  9. 2024Weakly-Supervised Anomaly Detection in Surveillance Videos Based on Two-Stream I3D Convolution Networkcited by 0