# AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection

arXiv: https://arxiv.org/abs/2206.15476 · HTML: https://mlsanity.com/paper/2206.15476/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 67 references that are in the map.
- First posted on arXiv 30 Jun 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [PyOD: A Python Toolbox for Scalable Outlier Detection](https://mlsanity.com/paper/1901.01588/) (2019)
- [Geometric Dataset Distances via Optimal Transport](https://mlsanity.com/paper/2002.02923/) (2020)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [COPOD: Copula-Based Outlier Detection](https://mlsanity.com/paper/2009.09463/) (2020)
- [SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation](https://mlsanity.com/paper/2206.08367/) (2022)
- [Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection](https://mlsanity.com/paper/1802.09089/) (2018)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data](https://mlsanity.com/paper/2108.09020/) (2021)
- [Generative Adversarial Active Learning for Unsupervised Outlier Detection](https://mlsanity.com/paper/1809.10816/) (2018)
- [E-GraphSAGE: A Graph Neural Network based Intrusion Detection System for IoT](https://mlsanity.com/paper/2103.16329/) (2021)
- [A Survey of Network-based Intrusion Detection Data Sets](https://mlsanity.com/paper/1903.02460/) (2019)
- [NetFlow Datasets for Machine Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2011.09144/) (2020)
- [LUNAR: Unifying Local Outlier Detection Methods via Graph Neural Networks](https://mlsanity.com/paper/2112.05355/) (2021)
- [Towards a Standard Feature Set for Network Intrusion Detection System Datasets](https://mlsanity.com/paper/2101.11315/) (2021)
- [Tree-based Intelligent Intrusion Detection System in Internet of Vehicles](https://mlsanity.com/paper/1910.08635/) (2019)
- [ZYELL-NCTU NetTraffic-1.0: A Large-Scale Dataset for Real-World Network Anomaly Detection](https://mlsanity.com/paper/2103.05767/) (2021)

## Built on by (most cited first, 11 of 11)

- [LAMDA: A Longitudinal Android Malware Benchmark for Concept Drift Analysis](https://mlsanity.com/paper/2505.18551/) (2025)
- [Chimera: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulation](https://mlsanity.com/paper/2508.07745/) (2025)
- [Env-Aware Anomaly Detection: Ignore Style Changes, Stay True to Content!](https://mlsanity.com/paper/2210.03103/) (2022)
- [Zero-Shot Anomaly Detection via Batch Normalization](https://mlsanity.com/paper/2302.07849/) (2023)
- [Robust Novelty Detection through Style-Conscious Feature Ranking](https://mlsanity.com/paper/2310.03738/) (2023)
- [Feature Distribution Shift Mitigation with Contrastive Pretraining for Intrusion Detection](https://mlsanity.com/paper/2404.15382/) (2024)
- [Shift Detection and Adaptation for Network Intrusion Detection](https://mlsanity.com/paper/2508.15100/) (2025)
- [When Normality Shifts: Risk-Aware Test-Time Adaptation for Unsupervised Tabular Anomaly Detection](https://mlsanity.com/paper/2605.10242/) (2026)
- [SEED: Semi-supervised Continual MalwarE Detection for Tackling ConcEpt Drift on a BuDget](https://mlsanity.com/paper/2605.24903/) (2026)
- [Rethinking Weak Supervision in Anomaly Detection: A Comprehensive Benchmark](https://mlsanity.com/paper/2605.26068/) (2026)
- [Towards Principled Continual Anomaly Detection: A Systematic Framework and Benchmark Scenarios](https://mlsanity.com/paper/2607.18289/) (2026)

## Abstract (arXiv)

Analyzing the distribution shift of data is a growing research direction in nowadays Machine Learning (ML), leading to emerging new benchmarks that focus on providing a suitable scenario for studying the generalization properties of ML models. The existing benchmarks are focused on supervised learning, and to the best of our knowledge, there is none for unsupervised learning. Therefore, we introduce an unsupervised anomaly detection benchmark with data that shifts over time, built over Kyoto-2006+, a traffic dataset for network intrusion detection. This type of data meets the premise of shifting the input distribution: it covers a large time span (10 years), with naturally occurring changes over time (eg users modifying their behavior patterns, and software updates). We first highlight the non-stationary nature of the data, using a basic per-feature analysis, t-SNE, and an Optimal Transport approach for measuring the overall distribution distances between years. Next, we propose AnoShift, a protocol splitting the data in IID, NEAR, and FAR testing splits. We validate the performance degradation over time with diverse models, ranging from classical approaches to deep learning. Finally, we show that by acknowledging the distribution shift problem and properly addressing it, the performance can be improved compared to the classical training which assumes independent and identically distributed data (on average, by up to 3% for our approach). Dataset and code are available at https://github.com/bit-ml/AnoShift/.