A Problem-Oriented Taxonomy of Evaluation Metrics for Time Series Anomaly Detection
Not on the map. Why?
Abstract (arXiv)
Time series anomaly detection is widely used in IoT and cyber-physical systems, yet its evaluation remains challenging due to diverse application objectives and heterogeneous metric assumptions. This study introduces a problem-oriented framework that reinterprets existing metrics based on the specific evaluation challenges they are designed to address, rather than their mathematical forms or output structures. We categorize over twenty commonly used metrics into six dimensions: 1) basic accuracy-driven evaluation; 2) timeliness-aware reward mechanisms; 3) tolerance to labeling imprecision; 4) penalties reflecting human-audit cost; 5) robustness against random or inflated scores; and 6) parameter-free comparability for cross-dataset benchmarking. Comprehensive experiments are conducted to examine metric behavior under genuine, random, and oracle detection scenarios. By comparing their resulting score distributions, we quantify each metric's discriminative ability -- its capability to distinguish meaningful detections from random noise. The results show that while most event-level metrics exhibit strong separability, several widely used metrics (e.g., NAB, Point-Adjust) demonstrate limited resistance to random-score inflation. These findings reveal that metric suitability must be inherently task-dependent and aligned with the operational objectives of IoT applications. The proposed framework offers a unified analytical perspective for understanding existing metrics and provides practical guidance for selecting or developing more context-aware, robust, and fair evaluation methodologies for time series anomaly detection.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 84 references that are in the map. First posted on arXiv 24 Nov 2025.
Builds on
29 of 84 references.
References · 29
- 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
- 2018Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholdingcited by 407
- 2021Graph Neural Network-Based Anomaly Detection in Multivariate Time Seriescited by 300
- 2017A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencodercited by 209
- 2020Multivariate Time-series Anomaly Detection via Graph Attention Networkcited by 165
- 2021LogBERT: Log Anomaly Detection via BERTcited by 114
- 2021Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoTcited by 104
- 2021Towards a Rigorous Evaluation of Time-series Anomaly Detectioncited by 95
- 2015Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmarkcited by 87
- 2022Local Evaluation of Time Series Anomaly Detection Algorithmscited by 79
- 2021An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Seriescited by 68
- 2018Precision and Recall for Time Seriescited by 67
17 more
- 2020Exathlon: A Benchmark for Explainable Anomaly Detection over Time Seriescited by 56
- 2020TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networkscited by 55
- 2022Graph-Augmented Normalizing Flows for Anomaly Detection of Multiple Time Seriescited by 42
- 2021Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signalscited by 41
- 2017Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networkscited by 40
- 2022Calibrated One-class Classification for Unsupervised Time Series Anomaly Detectioncited by 39
- 2021Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles -- Extended Versioncited by 35
- 2021Time Series Anomaly Detection for Cyber-Physical Systems via Neural System Identification and Bayesian Filteringcited by 31
- 2020TAnoGAN: Time Series Anomaly Detection with Generative Adversarial Networkscited by 29
- 2023Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Seriescited by 24
- 2020Multi-Scale One-Class Recurrent Neural Networks for Discrete Event Sequence Anomaly Detectioncited by 19
- 2020MSTREAM: Fast Anomaly Detection in Multi-Aspect Streamscited by 17
- 2022Are we certain it's anomalous?cited by 13
- 2024PATE: Proximity-Aware Time series anomaly Evaluationcited by 13
- 2021Time Series Anomaly Detection for Smart Grids: A Surveycited by 12
- 2020Statistical Evaluation of Anomaly Detectors for Sequencescited by 6
- 2022Anomaly Detection on Financial Time Series by Principal Component Analysis and Neural Networkscited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.