# Sequential Anomaly Detection using Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2004.10398 · HTML: https://mlsanity.com/paper/2004.10398/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 15 of its 61 references that are in the map.
- First posted on arXiv 22 Apr 2020.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection](https://mlsanity.com/paper/1607.00148/) (2016)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [Deep-Anomaly: Fully Convolutional Neural Network for Fast Anomaly Detection in Crowded Scenes](https://mlsanity.com/paper/1609.00866/) (2016)
- [Crowd Counting with Decomposed Uncertainty](https://mlsanity.com/paper/1903.07427/) (2019)
- [Recurrent Neural Radio Anomaly Detection](https://mlsanity.com/paper/1611.00301/) (2016)
- [Adaptive Pattern Matching with Reinforcement Learning for Dynamic Graphs](https://mlsanity.com/paper/1812.10321/) (2018)

## Built on by (most cited first, 20 of 20)

- [FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2206.06561/) (2022)
- [RLAD: Time Series Anomaly Detection through Reinforcement Learning and Active Learning](https://mlsanity.com/paper/2104.00543/) (2021)
- [A Survey on Deep Reinforcement Learning for Data Processing and Analytics](https://mlsanity.com/paper/2108.04526/) (2021)
- [Online Anomalous Subtrajectory Detection on Road Networks with Deep Reinforcement Learning](https://mlsanity.com/paper/2211.08415/) (2022)
- [Application of Deep Reinforcement Learning to Payment Fraud](https://mlsanity.com/paper/2112.04236/) (2021)
- [OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences](https://mlsanity.com/paper/2402.04567/) (2024)
- [Inverse reinforcement learning by expert imitation for the stochastic linear-quadratic optimal control problem](https://mlsanity.com/paper/2405.17085/) (2024)
- [Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlations](https://mlsanity.com/paper/2507.09742/) (2025)
- [CSCAD: Correlation Structure-based Collective Anomaly Detection in Complex System](https://mlsanity.com/paper/2105.14476/) (2021)
- [A Survey on Proactive Customer Care: Enabling Science and Steps to Realize it](https://mlsanity.com/paper/2110.05015/) (2021)
- [Deep Reinforcement Learning for Online Error Detection in Cyber-Physical Systems](https://mlsanity.com/paper/2302.01567/) (2023)
- [Learning Representative Trajectories of Dynamical Systems via Domain-Adaptive Imitation](https://mlsanity.com/paper/2304.10260/) (2023)
- [Anomalous Change Point Detection Using Probabilistic Predictive Coding](https://mlsanity.com/paper/2405.15727/) (2024)
- [EPR-GAIL: An EPR-Enhanced Hierarchical Imitation Learning Framework to Simulate Complex User Consumption Behaviors](https://mlsanity.com/paper/2503.06392/) (2025)
- [Anomalous Decision Discovery using Inverse Reinforcement Learning](https://mlsanity.com/paper/2507.04464/) (2025)
- [Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Space](https://mlsanity.com/paper/2508.15764/) (2025)
- [Intention-aware Hierarchical Diffusion Model for Long-term Trajectory Anomaly Detection](https://mlsanity.com/paper/2509.17068/) (2025)
- [A Context Engineering Framework for Improving Enterprise AI Agents based on Digital-Twin MDP](https://mlsanity.com/paper/2603.22083/) (2026)
- [LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection](https://mlsanity.com/paper/2606.08153/) (2026)
- [Rarity-Gated Context Conditioning for Offline Imitation Learning-Based Maritime Anomaly Detection](https://mlsanity.com/paper/2606.13311/) (2026)

## Abstract (arXiv)

One of the most interesting application scenarios in anomaly detection is when sequential data are targeted. For example, in a safety-critical environment, it is crucial to have an automatic detection system to screen the streaming data gathered by monitoring sensors and to report abnormal observations if detected in real-time. Oftentimes, stakes are much higher when these potential anomalies are intentional or goal-oriented. We propose an end-to-end framework for sequential anomaly detection using inverse reinforcement learning (IRL), whose objective is to determine the decision-making agent's underlying function which triggers his/her behavior. The proposed method takes the sequence of actions of a target agent (and possibly other meta information) as input. The agent's normal behavior is then understood by the reward function which is inferred via IRL. We use a neural network to represent a reward function. Using a learned reward function, we evaluate whether a new observation from the target agent follows a normal pattern. In order to construct a reliable anomaly detection method and take into consideration the confidence of the predicted anomaly score, we adopt a Bayesian approach for IRL. The empirical study on publicly available real-world data shows that our proposed method is effective in identifying anomalies.