# A Problem-Oriented Taxonomy of Evaluation Metrics for Time Series Anomaly Detection

arXiv: https://arxiv.org/abs/2511.18739 · HTML: https://mlsanity.com/paper/2511.18739/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 84 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 29 of 29)

- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [Graph Neural Network-Based Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2106.06947/) (2021)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [Multivariate Time-series Anomaly Detection via Graph Attention Network](https://mlsanity.com/paper/2009.02040/) (2020)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoT](https://mlsanity.com/paper/2104.03466/) (2021)
- [Towards a Rigorous Evaluation of Time-series Anomaly Detection](https://mlsanity.com/paper/2109.05257/) (2021)
- [Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark](https://mlsanity.com/paper/1510.03336/) (2015)
- [Local Evaluation of Time Series Anomaly Detection Algorithms](https://mlsanity.com/paper/2206.13167/) (2022)
- [An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series](https://mlsanity.com/paper/2109.11428/) (2021)
- [Precision and Recall for Time Series](https://mlsanity.com/paper/1803.03639/) (2018)
- [Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series](https://mlsanity.com/paper/2010.05073/) (2020)
- [TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks](https://mlsanity.com/paper/2009.07769/) (2020)
- [Graph-Augmented Normalizing Flows for Anomaly Detection of Multiple Time Series](https://mlsanity.com/paper/2202.07857/) (2022)
- [Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals](https://mlsanity.com/paper/2107.12626/) (2021)
- [Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks](https://mlsanity.com/paper/1710.09207/) (2017)
- [Calibrated One-class Classification for Unsupervised Time Series Anomaly Detection](https://mlsanity.com/paper/2207.12201/) (2022)
- [Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles -- Extended Version](https://mlsanity.com/paper/2111.11108/) (2021)
- [Time Series Anomaly Detection for Cyber-Physical Systems via Neural System Identification and Bayesian Filtering](https://mlsanity.com/paper/2106.07992/) (2021)
- [TAnoGAN: Time Series Anomaly Detection with Generative Adversarial Networks](https://mlsanity.com/paper/2008.09567/) (2020)
- [Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series](https://mlsanity.com/paper/2303.01272/) (2023)
- [Multi-Scale One-Class Recurrent Neural Networks for Discrete Event Sequence Anomaly Detection](https://mlsanity.com/paper/2008.13361/) (2020)
- [MSTREAM: Fast Anomaly Detection in Multi-Aspect Streams](https://mlsanity.com/paper/2009.08451/) (2020)
- [Are we certain it's anomalous?](https://mlsanity.com/paper/2211.09224/) (2022)
- [PATE: Proximity-Aware Time series anomaly Evaluation](https://mlsanity.com/paper/2405.12096/) (2024)
- [Time Series Anomaly Detection for Smart Grids: A Survey](https://mlsanity.com/paper/2107.08835/) (2021)
- [Statistical Evaluation of Anomaly Detectors for Sequences](https://mlsanity.com/paper/2008.05788/) (2020)
- [Anomaly Detection on Financial Time Series by Principal Component Analysis and Neural Networks](https://mlsanity.com/paper/2209.11686/) (2022)

## Built on by (most cited first, 1 of 1)

- [Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection](https://mlsanity.com/paper/2607.11969/) (2026)

## Abstract (arXiv)

Time series anomaly detection is widely used in IoT and cyber-physical systems, yet its evaluation remains challenging due to diverse application objectives and heterogeneous metric assumptions. This study introduces a problem-oriented framework that reinterprets existing metrics based on the specific evaluation challenges they are designed to address, rather than their mathematical forms or output structures. We categorize over twenty commonly used metrics into six dimensions: 1) basic accuracy-driven evaluation; 2) timeliness-aware reward mechanisms; 3) tolerance to labeling imprecision; 4) penalties reflecting human-audit cost; 5) robustness against random or inflated scores; and 6) parameter-free comparability for cross-dataset benchmarking. Comprehensive experiments are conducted to examine metric behavior under genuine, random, and oracle detection scenarios. By comparing their resulting score distributions, we quantify each metric's discriminative ability -- its capability to distinguish meaningful detections from random noise. The results show that while most event-level metrics exhibit strong separability, several widely used metrics (e.g., NAB, Point-Adjust) demonstrate limited resistance to random-score inflation. These findings reveal that metric suitability must be inherently task-dependent and aligned with the operational objectives of IoT applications. The proposed framework offers a unified analytical perspective for understanding existing metrics and provides practical guidance for selecting or developing more context-aware, robust, and fair evaluation methodologies for time series anomaly detection.