# Future Video Prediction from a Single Frame for Video Anomaly Detection

arXiv: https://arxiv.org/abs/2308.07783 · HTML: https://mlsanity.com/paper/2308.07783/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Latent Space Autoregression for Novelty Detection](https://mlsanity.com/paper/1807.01653/) (2018)
- [Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://mlsanity.com/paper/2011.07491/) (2020)
- [Object-centric Auto-encoders and Dummy Anomalies for Abnormal Event Detection in Video](https://mlsanity.com/paper/1812.04960/) (2018)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos](https://mlsanity.com/paper/1903.03295/) (2019)
- [Learning Normal Dynamics in Videos with Meta Prototype Network](https://mlsanity.com/paper/2104.06689/) (2021)
- [Video Generation from Single Semantic Label Map](https://mlsanity.com/paper/1903.04480/) (2019)
- [Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction](https://mlsanity.com/paper/2011.02763/) (2020)
- [Future Frame Prediction Using Convolutional VRNN for Anomaly Detection](https://mlsanity.com/paper/1909.02168/) (2019)
- [Dynamic Local Aggregation Network with Adaptive Clusterer for Anomaly Detection](https://mlsanity.com/paper/2207.10948/) (2022)
- [Anomaly Detection in Video Using Predictive Convolutional Long Short-Term Memory Networks](https://mlsanity.com/paper/1612.00390/) (2016)
- [Local Anomaly Detection in Videos using Object-Centric Adversarial Learning](https://mlsanity.com/paper/2011.06722/) (2020)
- [A Critical Study on the Recent Deep Learning Based Semi-Supervised Video Anomaly Detection Methods](https://mlsanity.com/paper/2111.01604/) (2021)
- [Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure](https://mlsanity.com/paper/2106.09051/) (2021)
- [Multi-Task Learning based Video Anomaly Detection with Attention](https://mlsanity.com/paper/2210.07697/) (2022)
- [Object Class Aware Video Anomaly Detection through Image Translation](https://mlsanity.com/paper/2205.01706/) (2022)

## Built on by (most cited first, 2 of 2)

- [Pix2Gif: Motion-Guided Diffusion for GIF Generation](https://mlsanity.com/paper/2403.04634/) (2024)
- [Mitral Regurgitation Recognition based on Unsupervised Out-of-Distribution Detection with Residual Diffusion Amplification](https://mlsanity.com/paper/2407.21497/) (2024)

## Abstract (arXiv)

Video anomaly detection (VAD) is an important but challenging task in computer vision. The main challenge rises due to the rarity of training samples to model all anomaly cases. Hence, semi-supervised anomaly detection methods have gotten more attention, since they focus on modeling normals and they detect anomalies by measuring the deviations from normal patterns. Despite impressive advances of these methods in modeling normal motion and appearance, long-term motion modeling has not been effectively explored so far. Inspired by the abilities of the future frame prediction proxy-task, we introduce the task of future video prediction from a single frame, as a novel proxy-task for video anomaly detection. This proxy-task alleviates the challenges of previous methods in learning longer motion patterns. Moreover, we replace the initial and future raw frames with their corresponding semantic segmentation map, which not only makes the method aware of object class but also makes the prediction task less complex for the model. Extensive experiments on the benchmark datasets (ShanghaiTech, UCSD-Ped1, and UCSD-Ped2) show the effectiveness of the method and the superiority of its performance compared to SOTA prediction-based VAD methods.