# A Generic Machine Learning Framework for Fully-Unsupervised Anomaly Detection with Contaminated Data

arXiv: https://arxiv.org/abs/2308.13352 · HTML: https://mlsanity.com/paper/2308.13352/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Classification-Based Anomaly Detection for General Data](https://mlsanity.com/paper/2005.02359/) (2020)
- [A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data](https://mlsanity.com/paper/1811.08055/) (2018)
- [Learning and Evaluating Representations for Deep One-class Classification](https://mlsanity.com/paper/2011.02578/) (2020)
- [MIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection](https://mlsanity.com/paper/1909.09347/) (2019)
- [Latent Outlier Exposure for Anomaly Detection with Contaminated Data](https://mlsanity.com/paper/2202.08088/) (2022)
- [Fusing Physics-based and Deep Learning Models for Prognostics](https://mlsanity.com/paper/2003.00732/) (2020)
- [Robust Anomaly Detection in Images using Adversarial Autoencoders](https://mlsanity.com/paper/1901.06355/) (2019)
- [Fully Learnable Deep Wavelet Transform for Unsupervised Monitoring of High-Frequency Time Series](https://mlsanity.com/paper/2105.00899/) (2021)
- [Self-supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection](https://mlsanity.com/paper/2106.06115/) (2021)
- [Detecting Anomalies within Time Series using Local Neural Transformations](https://mlsanity.com/paper/2202.03944/) (2022)
- [Unsupervised Learning of Anomaly Detection from Contaminated Image Data using Simultaneous Encoder Training](https://mlsanity.com/paper/1905.11034/) (2019)
- [Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection](https://mlsanity.com/paper/2207.11789/) (2022)

## Abstract (arXiv)

Anomaly detection (AD) tasks have been solved using machine learning algorithms in various domains and applications. The great majority of these algorithms use normal data to train a residual-based model and assign anomaly scores to unseen samples based on their dissimilarity with the learned normal regime. The underlying assumption of these approaches is that anomaly-free data is available for training. This is, however, often not the case in real-world operational settings, where the training data may be contaminated with an unknown fraction of abnormal samples. Training with contaminated data, in turn, inevitably leads to a deteriorated AD performance of the residual-based algorithms. In this paper we introduce a framework for a fully unsupervised refinement of contaminated training data for AD tasks. The framework is generic and can be applied to any residual-based machine learning model. We demonstrate the application of the framework to two public datasets of multivariate time series machine data from different application fields. We show its clear superiority over the naive approach of training with contaminated data without refinement. Moreover, we compare it to the ideal, unrealistic reference in which anomaly-free data would be available for training. The method is based on evaluating the contribution of individual samples to the generalization ability of a given model, and contrasting the contribution of anomalies with the one of normal samples. As a result, the proposed approach is comparable to, and often outperforms training with normal samples only.