# Making Reconstruction-based Method Great Again for Video Anomaly Detection

arXiv: https://arxiv.org/abs/2301.12048 · HTML: https://mlsanity.com/paper/2301.12048/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 28 Jan 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Latent Space Autoregression for Novelty Detection](https://mlsanity.com/paper/1807.01653/) (2018)
- [Anomaly Detection in Video Sequence with Appearance-Motion Correspondence](https://mlsanity.com/paper/1908.06351/) (2019)
- [Adversarially Learned Anomaly Detection](https://mlsanity.com/paper/1812.02288/) (2018)
- [Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos](https://mlsanity.com/paper/1903.03295/) (2019)
- [Abnormal Event Detection in Videos using Generative Adversarial Nets](https://mlsanity.com/paper/1708.09644/) (2017)
- [Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events](https://mlsanity.com/paper/2008.11988/) (2020)
- [ECACL: A Holistic Framework for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2104.09136/) (2021)
- [On Computation and Generalization of Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2001.02792/) (2020)
- [Future Frame Prediction Using Convolutional VRNN for Anomaly Detection](https://mlsanity.com/paper/1909.02168/) (2019)
- [Self-Directed Online Machine Learning for Topology Optimization](https://mlsanity.com/paper/2002.01927/) (2020)
- [HyperSTAR: Task-Aware Hyperparameters for Deep Networks](https://mlsanity.com/paper/2005.10524/) (2020)
- [SLA2P: Self-supervised Anomaly Detection with Adversarial Perturbation](https://mlsanity.com/paper/2111.12896/) (2021)

## Built on by (most cited first, 10 of 10)

- [VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding](https://mlsanity.com/paper/2312.02310/) (2023)
- [Towards Zero-shot 3D Anomaly Localization](https://mlsanity.com/paper/2412.04304/) (2024)
- [VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding](https://mlsanity.com/paper/2507.21507/) (2025)
- [Unveiling the Unseen: A Comprehensive Survey on Explainable Anomaly Detection in Images and Videos](https://mlsanity.com/paper/2302.06670/) (2023)
- [Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection](https://mlsanity.com/paper/2412.03044/) (2024)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [GRAM: An Interpretable Approach for Graph Anomaly Detection using Gradient Attention Maps](https://mlsanity.com/paper/2311.06153/) (2023)
- [Benchmarking Compact VLMs for Clip-Level Surveillance Anomaly Detection Under Weak Supervision](https://mlsanity.com/paper/2603.13306/) (2026)
- [A brief introduction to a framework named Multilevel Guidance-Exploration Network](https://mlsanity.com/paper/2312.04119/) (2023)
- [Forward Consistency Learning with Gated Context Aggregation for Video Anomaly Detection](https://mlsanity.com/paper/2601.18135/) (2026)

## Abstract (arXiv)

Anomaly detection in videos is a significant yet challenging problem. Previous approaches based on deep neural networks employ either reconstruction-based or prediction-based approaches. Nevertheless, existing reconstruction-based methods 1) rely on old-fashioned convolutional autoencoders and are poor at modeling temporal dependency; 2) are prone to overfit the training samples, leading to indistinguishable reconstruction errors of normal and abnormal frames during the inference phase. To address such issues, firstly, we get inspiration from transformer and propose textbf Spatio-textbf Temporal textbf Auto-textbf Trans-textbf Encoder, dubbed as STATE, as a new autoencoder model for enhanced consecutive frame reconstruction. Our STATE is equipped with a specifically designed learnable convolutional attention module for efficient temporal learning and reasoning. Secondly, we put forward a novel reconstruction-based input perturbation technique during testing to further differentiate anomalous frames. With the same perturbation magnitude, the testing reconstruction error of the normal frames lowers more than that of the abnormal frames, which contributes to mitigating the overfitting problem of reconstruction. Owing to the high relevance of the frame abnormality and the objects in the frame, we conduct object-level reconstruction using both the raw frame and the corresponding optical flow patches. Finally, the anomaly score is designed based on the combination of the raw and motion reconstruction errors using perturbed inputs. Extensive experiments on benchmark video anomaly detection datasets demonstrate that our approach outperforms previous reconstruction-based methods by a notable margin, and achieves state-of-the-art anomaly detection performance consistently. The code is available at https://github.com/wyzjack/MRMGA4VAD.