# Learning Not to Reconstruct Anomalies

arXiv: https://arxiv.org/abs/2110.09742 · HTML: https://mlsanity.com/paper/2110.09742/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [Latent Space Autoregression for Novelty Detection](https://mlsanity.com/paper/1807.01653/) (2018)
- [Anomaly Detection in Video Sequence with Appearance-Motion Correspondence](https://mlsanity.com/paper/1908.06351/) (2019)
- [Old is Gold: Redefining the Adversarially Learned One-Class Classifier Training Paradigm](https://mlsanity.com/paper/2004.07657/) (2020)
- [Object-centric Auto-encoders and Dummy Anomalies for Abnormal Event Detection in Video](https://mlsanity.com/paper/1812.04960/) (2018)
- [Street Scene: A new dataset and evaluation protocol for video anomaly detection](https://mlsanity.com/paper/1902.05872/) (2019)
- [A Background-Agnostic Framework with Adversarial Training for Abnormal Event Detection in Video](https://mlsanity.com/paper/2008.12328/) (2020)
- [Abnormal Event Detection in Videos using Generative Adversarial Nets](https://mlsanity.com/paper/1708.09644/) (2017)
- [CLAWS: Clustering Assisted Weakly Supervised Learning with Normalcy Suppression for Anomalous Event Detection](https://mlsanity.com/paper/2011.12077/) (2020)
- [Learning Deep Representations of Appearance and Motion for Anomalous Event Detection](https://mlsanity.com/paper/1510.01553/) (2015)
- [Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events](https://mlsanity.com/paper/2008.11988/) (2020)
- [Few-shot Scene-adaptive Anomaly Detection](https://mlsanity.com/paper/2007.07843/) (2020)
- [Joint Detection and Recounting of Abnormal Events by Learning Deep Generic Knowledge](https://mlsanity.com/paper/1709.09121/) (2017)
- [Continual Learning for Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/2004.07941/) (2020)
- [G2D: Generate to Detect Anomaly](https://mlsanity.com/paper/2006.11629/) (2020)
- [Any-Shot Sequential Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/2004.02072/) (2020)
- [Learning a distance function with a Siamese network to localize anomalies in videos](https://mlsanity.com/paper/2001.09189/) (2020)
- [Detecting abnormal events in video using Narrowed Normality Clusters](https://mlsanity.com/paper/1801.05030/) (2018)
- [Future Frame Prediction Using Convolutional VRNN for Anomaly Detection](https://mlsanity.com/paper/1909.02168/) (2019)
- [A Discriminative Framework for Anomaly Detection in Large Videos](https://mlsanity.com/paper/1609.08938/) (2016)
- [STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection](https://mlsanity.com/paper/1804.08381/) (2018)
- [Autoencoding Binary Classifiers for Supervised Anomaly Detection](https://mlsanity.com/paper/1903.10709/) (2019)

## Built on by (most cited first, 30 of 31)

- [Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection](https://mlsanity.com/paper/2111.09099/) (2021)
- [Generative Cooperative Learning for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2203.03962/) (2022)
- [Video Anomaly Detection by Solving Decoupled Spatio-Temporal Jigsaw Puzzles](https://mlsanity.com/paper/2207.10172/) (2022)
- [SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection](https://mlsanity.com/paper/2207.08003/) (2022)
- [Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors](https://mlsanity.com/paper/2306.12041/) (2023)
- [Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection](https://mlsanity.com/paper/2209.12148/) (2022)
- [Dynamic Local Aggregation Network with Adaptive Clusterer for Anomaly Detection](https://mlsanity.com/paper/2207.10948/) (2022)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [Cross-Domain Video Anomaly Detection without Target Domain Adaptation](https://mlsanity.com/paper/2212.07010/) (2022)
- [Clustering Aided Weakly Supervised Training to Detect Anomalous Events in Surveillance Videos](https://mlsanity.com/paper/2203.13704/) (2022)
- [Video Anomaly Detection via Spatio-Temporal Pseudo-Anomaly Generation : A Unified Approach](https://mlsanity.com/paper/2311.16514/) (2023)
- [Autoencoders for Anomaly Detection are Unreliable](https://mlsanity.com/paper/2501.13864/) (2025)
- [PseudoBound: Limiting the anomaly reconstruction capability of one-class classifiers using pseudo anomalies](https://mlsanity.com/paper/2303.10704/) (2023)
- [VideoPatchCore: An Effective Method to Memorize Normality for Video Anomaly Detection](https://mlsanity.com/paper/2409.16225/) (2024)
- [Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection](https://mlsanity.com/paper/2412.03044/) (2024)
- [SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Model](https://mlsanity.com/paper/2504.10320/) (2025)
- [Reconstruction Error-based Anomaly Detection with Few Outlying Examples](https://mlsanity.com/paper/2305.10464/) (2023)
- [Exploiting Autoencoder's Weakness to Generate Pseudo Anomalies](https://mlsanity.com/paper/2405.05886/) (2024)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation](https://mlsanity.com/paper/2211.15597/) (2022)
- [Rethinking Metrics and Benchmarks of Video Anomaly Detection](https://mlsanity.com/paper/2505.19022/) (2025)
- [Targeted Augmented Data for Audio Deepfake Detection](https://mlsanity.com/paper/2407.07598/) (2024)
- [Constricting Normal Latent Space for Anomaly Detection with Normal-only Training Data](https://mlsanity.com/paper/2403.16270/) (2024)
- [Fast reconstruction-based ROI triggering via anomaly detection in the CYGNO optical TPC](https://mlsanity.com/paper/2512.24290/) (2025)
- [Synthetic Pseudo Anomalies for Unsupervised Video Anomaly Detection: A Simple yet Efficient Framework based on Masked Autoencoder](https://mlsanity.com/paper/2303.05112/) (2023)
- [Exploring the Magnitude-Shape Plot Framework for Anomaly Detection in Crowded Video Scenes](https://mlsanity.com/paper/2412.20363/) (2024)
- [Video Anomaly Detection with Contours -- A Study](https://mlsanity.com/paper/2503.19588/) (2025)
- [GMFVAD: Using Grained Multi-modal Feature to Improve Video Anomaly Detection](https://mlsanity.com/paper/2510.20268/) (2025)
- [Semantic-Deviation-Anchored Multi-Branch Fusion for Unsupervised Anomaly Detection and Localization in Unstructured Conveyor-Belt Coal Scenes](https://mlsanity.com/paper/2602.07694/) (2026)
- [Look Inside Each Video: Rethinking How Video Anomaly Detection Is Evaluated](https://mlsanity.com/paper/2608.21854/) (2026)

## Abstract (arXiv)

Video anomaly detection is often seen as one-class classification (OCC) problem due to the limited availability of anomaly examples. Typically, to tackle this problem, an autoencoder (AE) is trained to reconstruct the input with training set consisting only of normal data. At test time, the AE is then expected to well reconstruct the normal data while poorly reconstructing the anomalous data. However, several studies have shown that, even with only normal data training, AEs can often start reconstructing anomalies as well which depletes the anomaly detection performance. To mitigate this problem, we propose a novel methodology to train AEs with the objective of reconstructing only normal data, regardless of the input (i.e., normal or abnormal). Since no real anomalies are available in the OCC settings, the training is assisted by pseudo anomalies that are generated by manipulating normal data to simulate the out-of-normal-data distribution. We additionally propose two ways to generate pseudo anomalies: patch and skip frame based. Extensive experiments on three challenging video anomaly datasets demonstrate the effectiveness of our method in improving conventional AEs, achieving state-of-the-art performance.