# Video Event Restoration Based on Keyframes for Video Anomaly Detection

arXiv: https://arxiv.org/abs/2304.05112 · HTML: https://mlsanity.com/paper/2304.05112/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks](https://mlsanity.com/paper/1710.01992/) (2017)
- [Anomaly Detection in Video Sequence with Appearance-Motion Correspondence](https://mlsanity.com/paper/1908.06351/) (2019)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)
- [Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos](https://mlsanity.com/paper/1903.03295/) (2019)
- [Learning Normal Dynamics in Videos with Meta Prototype Network](https://mlsanity.com/paper/2104.06689/) (2021)
- [Online Video Deblurring via Dynamic Temporal Blending Network](https://mlsanity.com/paper/1704.03285/) (2017)
- [Space-Time-Aware Multi-Resolution Video Enhancement](https://mlsanity.com/paper/2003.13170/) (2020)
- [Unmasking the abnormal events in video](https://mlsanity.com/paper/1705.08182/) (2017)
- [Unsupervised Deep Video Denoising](https://mlsanity.com/paper/2011.15045/) (2020)
- [RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2203.14186/) (2022)
- [Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction](https://mlsanity.com/paper/2011.02763/) (2020)
- [Progressive Temporal Feature Alignment Network for Video Inpainting](https://mlsanity.com/paper/2104.03507/) (2021)
- [Dynamic Local Aggregation Network with Adaptive Clusterer for Anomaly Detection](https://mlsanity.com/paper/2207.10948/) (2022)
- [Real-Time Anomaly Detection and Localization in Crowded Scenes](https://mlsanity.com/paper/1511.06936/) (2015)
- [AIVC: Artificial Intelligence based Video Codec](https://mlsanity.com/paper/2202.04365/) (2022)

## Built on by (most cited first, 30 of 54)

- [VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2308.11681/) (2023)
- [Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity](https://mlsanity.com/paper/2412.06171/) (2024)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [Open-Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2311.07042/) (2023)
- [Advancing Video Anomaly Detection: A Concise Review and a New Dataset](https://mlsanity.com/paper/2402.04857/) (2024)
- [Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2404.08531/) (2024)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Model](https://mlsanity.com/paper/2307.12545/) (2023)
- [Open-World Semantic Segmentation Including Class Similarity](https://mlsanity.com/paper/2403.07532/) (2024)
- [PANDA: Towards Generalist Video Anomaly Detection via Agentic AI Engineer](https://mlsanity.com/paper/2509.26386/) (2025)
- [Video Anomaly Detection in 10 Years: A Survey and Outlook](https://mlsanity.com/paper/2405.19387/) (2024)
- [AnyAnomaly: Zero-Shot Customizable Video Anomaly Detection with LVLM](https://mlsanity.com/paper/2503.04504/) (2025)
- [Anomize: Better Open Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2503.18094/) (2025)
- [Modeling Multiple Normal Action Representations for Error Detection in Procedural Tasks](https://mlsanity.com/paper/2503.22405/) (2025)
- [Context-aware Video Anomaly Detection in Long-Term Datasets](https://mlsanity.com/paper/2404.07887/) (2024)
- [Privacy-Preserving Video Anomaly Detection: A Survey](https://mlsanity.com/paper/2411.14565/) (2024)
- [Video Anomaly Detection via Spatio-Temporal Pseudo-Anomaly Generation : A Unified Approach](https://mlsanity.com/paper/2311.16514/) (2023)
- [A Physical Coherence Benchmark for Evaluating Video Generation Models via Optical Flow-guided Frame Prediction](https://mlsanity.com/paper/2502.05503/) (2025)
- [Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detection](https://mlsanity.com/paper/2503.03562/) (2025)
- [STNMamba: Mamba-based Spatial-Temporal Normality Learning for Video Anomaly Detection](https://mlsanity.com/paper/2412.20084/) (2024)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [VideoPatchCore: An Effective Method to Memorize Normality for Video Anomaly Detection](https://mlsanity.com/paper/2409.16225/) (2024)
- [ComplexVAD: Detecting Interaction Anomalies in Video](https://mlsanity.com/paper/2501.09733/) (2025)
- [VADMamba: Exploring State Space Models for Fast Video Anomaly Detection](https://mlsanity.com/paper/2503.21169/) (2025)
- [Foundation Models and Transformers for Anomaly Detection: A Survey](https://mlsanity.com/paper/2507.15905/) (2025)
- [No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection](https://mlsanity.com/paper/2602.19248/) (2026)
- [GlanceVAD: Exploring Glance Supervision for Label-efficient Video Anomaly Detection](https://mlsanity.com/paper/2403.06154/) (2024)
- [AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection](https://mlsanity.com/paper/2504.04495/) (2025)
- [Vision-Language Models Assisted Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2409.14109/) (2024)
- [Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection](https://mlsanity.com/paper/2412.03044/) (2024)

## Abstract (arXiv)

Video anomaly detection (VAD) is a significant computer vision problem. Existing deep neural network (DNN) based VAD methods mostly follow the route of frame reconstruction or frame prediction. However, the lack of mining and learning of higher-level visual features and temporal context relationships in videos limits the further performance of these two approaches. Inspired by video codec theory, we introduce a brand-new VAD paradigm to break through these limitations: First, we propose a new task of video event restoration based on keyframes. Encouraging DNN to infer missing multiple frames based on video keyframes so as to restore a video event, which can more effectively motivate DNN to mine and learn potential higher-level visual features and comprehensive temporal context relationships in the video. To this end, we propose a novel U-shaped Swin Transformer Network with Dual Skip Connections (USTN-DSC) for video event restoration, where a cross-attention and a temporal upsampling residual skip connection are introduced to further assist in restoring complex static and dynamic motion object features in the video. In addition, we propose a simple and effective adjacent frame difference loss to constrain the motion consistency of the video sequence. Extensive experiments on benchmarks demonstrate that USTN-DSC outperforms most existing methods, validating the effectiveness of our method.