# RLAD: Time Series Anomaly Detection through Reinforcement Learning and Active Learning

arXiv: https://arxiv.org/abs/2104.00543 · HTML: https://mlsanity.com/paper/2104.00543/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Deep Semi-Supervised Anomaly Detection](https://mlsanity.com/paper/1906.02694/) (2019)
- [Semi-Supervised Learning with Ladder Networks](https://mlsanity.com/paper/1507.02672/) (2015)
- [Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications](https://mlsanity.com/paper/1802.03903/) (2018)
- [Time-Series Anomaly Detection Service at Microsoft](https://mlsanity.com/paper/1906.03821/) (2019)
- [Deep Anomaly Detection with Deviation Networks](https://mlsanity.com/paper/1911.08623/) (2019)
- [Good Semi-supervised Learning that Requires a Bad GAN](https://mlsanity.com/paper/1705.09783/) (2017)
- [Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection](https://mlsanity.com/paper/1806.04808/) (2018)
- [Toward Supervised Anomaly Detection](https://mlsanity.com/paper/1401.6424/) (2014)
- [Sequential Anomaly Detection using Inverse Reinforcement Learning](https://mlsanity.com/paper/2004.10398/) (2020)
- [An unsupervised spatiotemporal graphical modeling approach to anomaly detection in distributed CPS](https://mlsanity.com/paper/1512.07876/) (2015)

## Built on by (most cited first, 13 of 13)

- [Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles](https://mlsanity.com/paper/2207.12380/) (2022)
- [Task-Driven Detection of Distribution Shifts with Statistical Guarantees for Robot Learning](https://mlsanity.com/paper/2106.13703/) (2021)
- [Anomaly Detection in Time Series Data Using Reinforcement Learning, Variational Autoencoder, and Active Learning](https://mlsanity.com/paper/2504.02999/) (2025)
- [Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies](https://mlsanity.com/paper/2507.06519/) (2025)
- [Towards Dynamic Trend Filtering through Trend Point Detection with Reinforcement Learning](https://mlsanity.com/paper/2406.03665/) (2024)
- [Label-Efficient Interactive Time-Series Anomaly Detection](https://mlsanity.com/paper/2212.14621/) (2022)
- [Infrastructure-based End-to-End Learning and Prevention of Driver Failure](https://mlsanity.com/paper/2303.12224/) (2023)
- [Tackling Diverse Minorities in Imbalanced Classification](https://mlsanity.com/paper/2308.14838/) (2023)
- [Expert enhanced dynamic time warping based anomaly detection](https://mlsanity.com/paper/2310.02280/) (2023)
- [An Active Learning Framework with a Class Balancing Strategy for Time Series Classification](https://mlsanity.com/paper/2405.12122/) (2024)
- [DRTA: Dynamic Reward Scaling for Reinforcement Learning in Time Series Anomaly Detection](https://mlsanity.com/paper/2508.18474/) (2025)
- [Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection: A VAE-Enhanced Reinforcement Learning Approach](https://mlsanity.com/paper/2511.12351/) (2025)
- [BoundAD: Boundary-Aware Negative Generation for Time Series Anomaly Detection](https://mlsanity.com/paper/2603.18111/) (2026)

## Abstract (arXiv)

We introduce a new semi-supervised, time series anomaly detection algorithm that uses deep reinforcement learning (DRL) and active learning to efficiently learn and adapt to anomalies in real-world time series data. Our model - called RLAD - makes no assumption about the underlying mechanism that produces the observation sequence and continuously adapts the detection model based on experience with anomalous patterns. In addition, it requires no manual tuning of parameters and outperforms all state-of-art methods we compare with, both unsupervised and semi-supervised, across several figures of merit. More specifically, we outperform the best unsupervised approach by a factor of 1.58 on the F1 score, with only 1% of labels and up to around 4.4x on another real-world dataset with only 0.1% of labels. We compare RLAD with seven deep-learning based algorithms across two common anomaly detection datasets with up to around 3M data points and between 0.28% to 2.65% anomalies.We outperform all of them across several important performance metrics.