# A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction

arXiv: https://arxiv.org/abs/2108.06852 · HTML: https://mlsanity.com/paper/2108.06852/

## Key facts

- Cited by 129 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 55 references that are in the map.
- First posted on arXiv 16 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [A Variational U-Net for Conditional Appearance and Shape Generation](https://mlsanity.com/paper/1804.04694/) (2018)
- [Anomaly Detection in Video Sequence with Appearance-Motion Correspondence](https://mlsanity.com/paper/1908.06351/) (2019)
- [Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos](https://mlsanity.com/paper/1903.03295/) (2019)
- [Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events](https://mlsanity.com/paper/2008.11988/) (2020)
- [ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal](https://mlsanity.com/paper/1908.01323/) (2019)
- [A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2101.00545/) (2021)
- [RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal](https://mlsanity.com/paper/1911.09178/) (2019)
- [Future Frame Prediction Using Convolutional VRNN for Anomaly Detection](https://mlsanity.com/paper/1909.02168/) (2019)
- [Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning](https://mlsanity.com/paper/2108.02366/) (2021)
- [Video Anomaly Detection and Localization via Gaussian Mixture Fully Convolutional Variational Autoencoder](https://mlsanity.com/paper/1805.11223/) (2018)
- [CRD-CGAN: Category-Consistent and Relativistic Constraints for Diverse Text-to-Image Generation](https://mlsanity.com/paper/2107.13516/) (2021)
- [Decoupled Appearance and Motion Learning for Efficient Anomaly Detection in Surveillance Video](https://mlsanity.com/paper/2011.05054/) (2020)

## Built on by (most cited first, 30 of 129)

- [Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection](https://mlsanity.com/paper/2111.09099/) (2021)
- [UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection](https://mlsanity.com/paper/2111.08644/) (2021)
- [Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2302.05160/) (2023)
- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [Harnessing Large Language Models for Training-free Video Anomaly Detection](https://mlsanity.com/paper/2404.01014/) (2024)
- [Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity](https://mlsanity.com/paper/2412.06171/) (2024)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [Video Anomaly Detection by Solving Decoupled Spatio-Temporal Jigsaw Puzzles](https://mlsanity.com/paper/2207.10172/) (2022)
- [Normalizing Flows for Human Pose Anomaly Detection](https://mlsanity.com/paper/2211.10946/) (2022)
- [Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models](https://mlsanity.com/paper/2302.05087/) (2023)
- [Open-Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2311.07042/) (2023)
- [Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2407.10299/) (2024)
- [Exploiting Completeness and Uncertainty of Pseudo Labels for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2212.04090/) (2022)
- [SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection](https://mlsanity.com/paper/2207.08003/) (2022)
- [Learning Prompt-Enhanced Context Features for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2306.14451/) (2023)
- [A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipation](https://mlsanity.com/paper/2305.13611/) (2023)
- [Diversity-Measurable Anomaly Detection](https://mlsanity.com/paper/2303.05047/) (2023)
- [Hierarchical Semantic Contrast for Scene-aware Video Anomaly Detection](https://mlsanity.com/paper/2303.13051/) (2023)
- [Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors](https://mlsanity.com/paper/2306.12041/) (2023)
- [SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2111.13495/) (2021)
- [Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection](https://mlsanity.com/paper/2209.12148/) (2022)
- [Diverse Human Motion Prediction via Gumbel-Softmax Sampling from an Auxiliary Space](https://mlsanity.com/paper/2207.07351/) (2022)
- [Dynamic Local Aggregation Network with Adaptive Clusterer for Anomaly Detection](https://mlsanity.com/paper/2207.10948/) (2022)
- [Delving into CLIP latent space for Video Anomaly Recognition](https://mlsanity.com/paper/2310.02835/) (2023)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2312.02530/) (2023)
- [Regularity Learning via Explicit Distribution Modeling for Skeletal Video Anomaly Detection](https://mlsanity.com/paper/2112.03649/) (2021)
- [BatchNorm-based Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2311.15367/) (2023)
- [MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection](https://mlsanity.com/paper/2403.14497/) (2024)
- [Networking Systems for Video Anomaly Detection: A Tutorial and Survey](https://mlsanity.com/paper/2405.10347/) (2024)

## Abstract (arXiv)

In this paper, we propose HF2-VAD, a Hybrid framework that integrates Flow reconstruction and Frame prediction seamlessly to handle Video Anomaly Detection. Firstly, we design the network of ML-MemAE-SC (Multi-Level Memory modules in an Autoencoder with Skip Connections) to memorize normal patterns for optical flow reconstruction so that abnormal events can be sensitively identified with larger flow reconstruction errors. More importantly, conditioned on the reconstructed flows, we then employ a Conditional Variational Autoencoder (CVAE), which captures the high correlation between video frame and optical flow, to predict the next frame given several previous frames. By CVAE, the quality of flow reconstruction essentially influences that of frame prediction. Therefore, poorly reconstructed optical flows of abnormal events further deteriorate the quality of the final predicted future frame, making the anomalies more detectable. Experimental results demonstrate the effectiveness of the proposed method. Code is available at hrefhttps://github.com/LiUzHiAn/hf2vadhttps://github.com/LiUzHiAn/hf2vad.