# Unsupervised Model Selection for Time-series Anomaly Detection

arXiv: https://arxiv.org/abs/2210.01078 · HTML: https://mlsanity.com/paper/2210.01078/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 70 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Time Series Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2002.12478/) (2020)
- [Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications](https://mlsanity.com/paper/1802.03903/) (2018)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [A review on outlier/anomaly detection in time series data](https://mlsanity.com/paper/2002.04236/) (2020)
- [Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress](https://mlsanity.com/paper/2009.13807/) (2020)
- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Dilated Recurrent Neural Networks](https://mlsanity.com/paper/1710.02224/) (2017)
- [Model Selection in Contextual Stochastic Bandit Problems](https://mlsanity.com/paper/2003.01704/) (2020)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [Unsupervised Model Selection for Variational Disentangled Representation Learning](https://mlsanity.com/paper/1905.12614/) (2019)
- [Neural Contextual Anomaly Detection for Time Series](https://mlsanity.com/paper/2107.07702/) (2021)
- [How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms?](https://mlsanity.com/paper/1607.01152/) (2016)
- [Deep Generative model with Hierarchical Latent Factors for Time Series Anomaly Detection](https://mlsanity.com/paper/2202.07586/) (2022)
- [Online false discovery rate control for anomaly detection in time series](https://mlsanity.com/paper/2112.03196/) (2021)
- [Automated Model Selection for Time-Series Anomaly Detection](https://mlsanity.com/paper/2009.04395/) (2020)
- [Concentric mixtures of Mallows models for top-k rankings: sampling and identifiability](https://mlsanity.com/paper/2010.14260/) (2020)
- [A Large-scale Study on Unsupervised Outlier Model Selection: Do Internal Strategies Suffice?](https://mlsanity.com/paper/2104.01422/) (2021)
- [Weak Supervision for Affordable Modeling of Electrocardiogram Data](https://mlsanity.com/paper/2201.02936/) (2022)
- [Classifying Unstructured Clinical Notes via Automatic Weak Supervision](https://mlsanity.com/paper/2206.12088/) (2022)
- [Time Series Anomaly Detection via Reinforcement Learning-Based Model Selection](https://mlsanity.com/paper/2205.09884/) (2022)
- [Self-supervised learning for fast and scalable time series hyper-parameter tuning](https://mlsanity.com/paper/2102.05740/) (2021)
- [Weakly Supervised Classification of Vital Sign Alerts as Real or Artifact](https://mlsanity.com/paper/2206.09074/) (2022)

## Built on by (most cited first, 25 of 25)

- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [TimeSeriesExam: A time series understanding exam](https://mlsanity.com/paper/2410.14752/) (2024)
- [MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2312.02530/) (2023)
- [Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection](https://mlsanity.com/paper/2401.15123/) (2024)
- [Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series](https://mlsanity.com/paper/2303.01272/) (2023)
- [Inherently Interpretable Time Series Classification via Multiple Instance Learning](https://mlsanity.com/paper/2311.10049/) (2023)
- [Unraveling the "Anomaly" in Time Series Anomaly Detection: A Self-supervised Tri-domain Solution](https://mlsanity.com/paper/2311.11235/) (2023)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [mTSBench: Benchmarking Multivariate Time Series Anomaly Detection and Model Selection at Scale](https://mlsanity.com/paper/2506.21550/) (2025)
- [Unlocking Pretrained Vision Transformers for Time Series Classification](https://mlsanity.com/paper/2506.08641/) (2025)
- [TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale](https://mlsanity.com/paper/2604.10291/) (2026)
- [ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response](https://mlsanity.com/paper/2604.21199/) (2026)
- [DELPHYNE: A Pre-Trained Model for General and Financial Time Series](https://mlsanity.com/paper/2506.06288/) (2025)
- [Structured Temporal Causality for Interpretable Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2510.16511/) (2025)
- [PUPAE: Intuitive and Actionable Explanations for Time Series Anomalies](https://mlsanity.com/paper/2401.09489/) (2024)
- [A Reliable Framework for Human-in-the-Loop Anomaly Detection in Time Series](https://mlsanity.com/paper/2405.03234/) (2024)
- [Refining Time Series Anomaly Detectors using Large Language Models](https://mlsanity.com/paper/2503.21833/) (2025)
- [MSAD: A Deep Dive into Model Selection for Time series Anomaly Detection](https://mlsanity.com/paper/2510.26643/) (2025)
- [RAMSeS: Robust and Adaptive Model Selection for Time-Series Anomaly Detection Algorithms](https://mlsanity.com/paper/2602.21766/) (2026)
- [Evaluation Strategy of Time-series Anomaly Detection with Decay Function](https://mlsanity.com/paper/2305.09691/) (2023)
- [Multivariate Time-series Anomaly Detection via Dynamic Model Pool & Ensembling](https://mlsanity.com/paper/2601.02037/) (2026)
- [IMPACT: Influence Modeling for Open-Set Time Series Anomaly Detection](https://mlsanity.com/paper/2603.29183/) (2026)
- [From Benchmarks to Production: Transferring Time Series Anomaly Detection Methods for Electricity Production Monitoring](https://mlsanity.com/paper/2609.39257/) (2026)
- [Retrieval Over Training: Similarity search-based Model Selection for Time Series Anomaly Detection](https://mlsanity.com/paper/2610.20243/) (2026)

## Abstract (arXiv)

Anomaly detection in time-series has a wide range of practical applications. While numerous anomaly detection methods have been proposed in the literature, a recent survey concluded that no single method is the most accurate across various datasets. To make matters worse, anomaly labels are scarce and rarely available in practice. The practical problem of selecting the most accurate model for a given dataset without labels has received little attention in the literature. This paper answers this question i.e. Given an unlabeled dataset and a set of candidate anomaly detectors, how can we select the most accurate model? To this end, we identify three classes of surrogate (unsupervised) metrics, namely, prediction error, model centrality, and performance on injected synthetic anomalies, and show that some metrics are highly correlated with standard supervised anomaly detection performance metrics such as the F1 score, but to varying degrees. We formulate metric combination with multiple imperfect surrogate metrics as a robust rank aggregation problem. We then provide theoretical justification behind the proposed approach. Large-scale experiments on multiple real-world datasets demonstrate that our proposed unsupervised approach is as effective as selecting the most accurate model based on partially labeled data.