# Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection: A VAE-Enhanced Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2511.12351 · HTML: https://mlsanity.com/paper/2511.12351/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 24 of 24)

- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data](https://mlsanity.com/paper/2201.07284/) (2022)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [Multivariate Time-series Anomaly Detection via Graph Attention Network](https://mlsanity.com/paper/2009.02040/) (2020)
- [Deep Learning for Anomaly Detection: A Review](https://mlsanity.com/paper/2007.02500/) (2020)
- [Deep Learning for Time Series Anomaly Detection: A Survey](https://mlsanity.com/paper/2211.05244/) (2022)
- [DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection](https://mlsanity.com/paper/2306.10347/) (2023)
- [AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme](https://mlsanity.com/paper/2305.04468/) (2023)
- [Unsupervised anomaly detection algorithms on real-world data: how many do we need?](https://mlsanity.com/paper/2305.00735/) (2023)
- [Adaptive Preference Scaling for Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2406.02764/) (2024)
- [Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment](https://mlsanity.com/paper/2503.18991/) (2025)
- [CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detection](https://mlsanity.com/paper/2308.09296/) (2023)
- [RLAD: Time Series Anomaly Detection through Reinforcement Learning and Active Learning](https://mlsanity.com/paper/2104.00543/) (2021)
- [Online false discovery rate control for anomaly detection in time series](https://mlsanity.com/paper/2112.03196/) (2021)
- [Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Values](https://mlsanity.com/paper/2401.05800/) (2024)
- [Online Multivariate Anomaly Detection and Localization for High-dimensional Settings](https://mlsanity.com/paper/1905.07107/) (2019)
- [Variational Autoencoder for Anomaly Detection: A Comparative Study](https://mlsanity.com/paper/2408.13561/) (2024)
- [Anomaly Detection for Industrial Applications, Its Challenges, Solutions, and Future Directions: A Review](https://mlsanity.com/paper/2501.11310/) (2025)
- [Cyber-Physical Systems Security: A Comprehensive Review of Anomaly Detection Techniques](https://mlsanity.com/paper/2502.13256/) (2025)
- [Anomaly Detection in Time Series Data Using Reinforcement Learning, Variational Autoencoder, and Active Learning](https://mlsanity.com/paper/2504.02999/) (2025)
- [Calibrated Unsupervised Anomaly Detection in Multivariate Time-series using Reinforcement Learning](https://mlsanity.com/paper/2502.03245/) (2025)

## Abstract (arXiv)

Detecting anomalies in multivariate time series is essential for monitoring complex industrial systems, where high dimensionality, limited labeled data, and subtle dependencies between sensors cause significant challenges. This paper presents a deep reinforcement learning framework that combines a Variational Autoencoder (VAE), an LSTM-based Deep Q-Network (DQN), dynamic reward shaping, and an active learning module to address these issues in a unified learning framework. The main contribution is the implementation of Dynamic Reward Scaling for Multivariate Time Series Anomaly Detection (DRSMT), which demonstrates how each component enhances the detection process. The VAE captures compact latent representations and reduces noise. The DQN enables adaptive, sequential anomaly classification, and the dynamic reward shaping balances exploration and exploitation during training by adjusting the importance of reconstruction and classification signals. In addition, active learning identifies the most uncertain samples for labeling, reducing the need for extensive manual supervision. Experiments on two multivariate benchmarks, namely Server Machine Dataset (SMD) and Water Distribution Testbed (WADI), show that the proposed method outperforms existing baselines in F1-score and AU-PR. These results highlight the effectiveness of combining generative modeling, reinforcement learning, and selective supervision for accurate and scalable anomaly detection in real-world multivariate systems.