# Anomaly Detection: How to Artificially Increase your F1-Score with a Biased Evaluation Protocol

arXiv: https://arxiv.org/abs/2106.16020 · HTML: https://mlsanity.com/paper/2106.16020/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 12 of 12)

- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training](https://mlsanity.com/paper/1805.06725/) (2018)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [DROCC: Deep Robust One-Class Classification](https://mlsanity.com/paper/2002.12718/) (2020)
- [Old is Gold: Redefining the Adversarially Learned One-Class Classifier Training Paradigm](https://mlsanity.com/paper/2004.07657/) (2020)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)
- [Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks](https://mlsanity.com/paper/1710.09207/) (2017)
- [Robust Subspace Recovery Layer for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.00152/) (2019)
- [GAN Ensemble for Anomaly Detection](https://mlsanity.com/paper/2012.07988/) (2020)
- [Robust Variational Autoencoders for Outlier Detection and Repair of Mixed-Type Data](https://mlsanity.com/paper/1907.06671/) (2019)
- [Memory Augmented Generative Adversarial Networks for Anomaly Detection](https://mlsanity.com/paper/2002.02669/) (2020)
- [Regularized Cycle Consistent Generative Adversarial Network for Anomaly Detection](https://mlsanity.com/paper/2001.06591/) (2020)

## Built on by (most cited first, 9 of 9)

- [Estimating the Contamination Factor's Distribution in Unsupervised Anomaly Detection](https://mlsanity.com/paper/2210.10487/) (2022)
- [A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithms](https://mlsanity.com/paper/2204.09825/) (2022)
- [Deep-Disaster: Unsupervised Disaster Detection and Localization Using Visual Data](https://mlsanity.com/paper/2202.00050/) (2022)
- [Unsupervised Anomaly Detection for Auditing Data and Impact of Categorical Encodings](https://mlsanity.com/paper/2210.14056/) (2022)
- [FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data](https://mlsanity.com/paper/2408.04442/) (2024)
- [Automatic Tuberculosis and COVID-19 cough classification using deep learning](https://mlsanity.com/paper/2205.05480/) (2022)
- [Benchmarking Anomaly Detection Algorithms: Deep Learning and Beyond](https://mlsanity.com/paper/2402.07281/) (2024)
- [Introducing 'Inside' Out of Distribution](https://mlsanity.com/paper/2407.04534/) (2024)
- [Weakly-Supervised Anomaly Detection in Surveillance Videos Based on Two-Stream I3D Convolution Network](https://mlsanity.com/paper/2411.08755/) (2024)

## Abstract (arXiv)

Anomaly detection is a widely explored domain in machine learning. Many models are proposed in the literature, and compared through different metrics measured on various datasets. The most popular metrics used to compare performances are F1-score, AUC and AVPR. In this paper, we show that F1-score and AVPR are highly sensitive to the contamination rate. One consequence is that it is possible to artificially increase their values by modifying the train-test split procedure. This leads to misleading comparisons between algorithms in the literature, especially when the evaluation protocol is not well detailed. Moreover, we show that the F1-score and the AVPR cannot be used to compare performances on different datasets as they do not reflect the intrinsic difficulty of modeling such data. Based on these observations, we claim that F1-score and AVPR should not be used as metrics for anomaly detection. We recommend a generic evaluation procedure for unsupervised anomaly detection, including the use of other metrics such as the AUC, which are more robust to arbitrary choices in the evaluation protocol.