# OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences

arXiv: https://arxiv.org/abs/2402.04567 · HTML: https://mlsanity.com/paper/2402.04567/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023)
- [LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection](https://mlsanity.com/paper/1607.00148/) (2016)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [The Predictron: End-To-End Learning and Planning](https://mlsanity.com/paper/1612.08810/) (2016)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.02926/) (2018)
- [How to Incorporate Monotonicity in Deep Networks While Preserving Flexibility?](https://mlsanity.com/paper/1909.10662/) (2019)
- [Sequential Anomaly Detection using Inverse Reinforcement Learning](https://mlsanity.com/paper/2004.10398/) (2020)
- [Rethinking ValueDice: Does It Really Improve Performance?](https://mlsanity.com/paper/2202.02468/) (2022)
- [Policy Entropy for Out-of-Distribution Classification](https://mlsanity.com/paper/2005.12069/) (2020)
- [Sequential Adversarial Anomaly Detection for One-Class Event Data](https://mlsanity.com/paper/1910.09161/) (2019)
- [A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/2109.09889/) (2021)
- [Monotonicity Regularization: Improved Penalties and Novel Applications to Disentangled Representation Learning and Robust Classification](https://mlsanity.com/paper/2205.08247/) (2022)

## Built on by (most cited first, 3 of 3)

- [Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts](https://mlsanity.com/paper/2412.04220/) (2024)
- [CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration](https://mlsanity.com/paper/2509.17458/) (2025)
- [Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Space](https://mlsanity.com/paper/2508.15764/) (2025)

## Abstract (arXiv)

Anomaly detection in decision-making sequences is a challenging problem due to the complexity of normality representation learning and the sequential nature of the task. Most existing methods based on Reinforcement Learning (RL) are difficult to implement in the real world due to unrealistic assumptions, such as having access to environment dynamics, reward signals, and online interactions with the environment. To address these limitations, we propose an unsupervised method named Offline Imitation Learning based Anomaly Detection (OIL-AD), which detects anomalies in decision-making sequences using two extracted behaviour features: action optimality and sequential association. Our offline learning model is an adaptation of behavioural cloning with a transformer policy network, where we modify the training process to learn a Q function and a state value function from normal trajectories. We propose that the Q function and the state value function can provide sufficient information about agents' behavioural data, from which we derive two features for anomaly detection. The intuition behind our method is that the action optimality feature derived from the Q function can differentiate the optimal action from others at each local state, and the sequential association feature derived from the state value function has the potential to maintain the temporal correlations between decisions (state-action pairs). Our experiments show that OIL-AD can achieve outstanding online anomaly detection performance with up to 34.8% improvement in F1 score over comparable baselines.