# Convolutional Transformer based Dual Discriminator Generative Adversarial Networks for Video Anomaly Detection

arXiv: https://arxiv.org/abs/2107.13720 · HTML: https://mlsanity.com/paper/2107.13720/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction](https://mlsanity.com/paper/1704.02971/) (2017)
- [A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data](https://mlsanity.com/paper/1811.08055/) (2018)
- [Abnormal Event Detection in Videos using Generative Adversarial Nets](https://mlsanity.com/paper/1708.09644/) (2017)
- [Abnormal Event Detection in Videos using Spatiotemporal Autoencoder](https://mlsanity.com/paper/1701.01546/) (2017)
- [Dual Discriminator Generative Adversarial Nets](https://mlsanity.com/paper/1709.03831/) (2017)

## Built on by (most cited first, 30 of 31)

- [CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2212.05136/) (2022)
- [Video Anomaly Detection by Solving Decoupled Spatio-Temporal Jigsaw Puzzles](https://mlsanity.com/paper/2207.10172/) (2022)
- [Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models](https://mlsanity.com/paper/2302.05087/) (2023)
- [A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipation](https://mlsanity.com/paper/2305.13611/) (2023)
- [Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts](https://mlsanity.com/paper/2408.05905/) (2024)
- [Hierarchical Semantic Contrast for Scene-aware Video Anomaly Detection](https://mlsanity.com/paper/2303.13051/) (2023)
- [Delving into CLIP latent space for Video Anomaly Recognition](https://mlsanity.com/paper/2310.02835/) (2023)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [Regularity Learning via Explicit Distribution Modeling for Skeletal Video Anomaly Detection](https://mlsanity.com/paper/2112.03649/) (2021)
- [Deep Anomaly Discovery From Unlabeled Videos via Normality Advantage and Self-Paced Refinement](https://mlsanity.com/paper/2108.01975/) (2021)
- [Networking Systems for Video Anomaly Detection: A Tutorial and Survey](https://mlsanity.com/paper/2405.10347/) (2024)
- [Applications of Generative Adversarial Networks in Anomaly Detection: A Systematic Literature Review](https://mlsanity.com/paper/2110.12076/) (2021)
- [MissionGNN: Hierarchical Multimodal GNN-based Weakly Supervised Video Anomaly Recognition with Mission-Specific Knowledge Graph Generation](https://mlsanity.com/paper/2406.18815/) (2024)
- [Anomize: Better Open Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2503.18094/) (2025)
- [Multi-Contextual Predictions with Vision Transformer for Video Anomaly Detection](https://mlsanity.com/paper/2206.08568/) (2022)
- [Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2302.08641/) (2023)
- [Foundation Models and Transformers for Anomaly Detection: A Survey](https://mlsanity.com/paper/2507.15905/) (2025)
- [Prompt-Driven Temporal Domain Adaptation for Nighttime UAV Tracking](https://mlsanity.com/paper/2409.18533/) (2024)
- [SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Model](https://mlsanity.com/paper/2504.10320/) (2025)
- [Exploiting Spatial-temporal Correlations for Video Anomaly Detection](https://mlsanity.com/paper/2211.00829/) (2022)
- [Interleaving One-Class and Weakly-Supervised Models with Adaptive Thresholding for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2401.13551/) (2024)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning](https://mlsanity.com/paper/2411.09072/) (2024)
- [TeG: Temporal-Granularity Method for Anomaly Detection with Attention in Smart City Surveillance](https://mlsanity.com/paper/2411.11003/) (2024)
- [Efficient Vision Language Model Fine-tuning for Text-based Person Anomaly Search](https://mlsanity.com/paper/2502.03230/) (2025)
- [Uncertainty-Weighted Fusion of Image and Synthetic Event for Video Anomaly Detection](https://mlsanity.com/paper/2505.02393/) (2025)
- [PicT: A Slim Weakly Supervised Vision Transformer for Pavement Distress Classification](https://mlsanity.com/paper/2209.10074/) (2022)
- [TASL-Net: Tri-Attention Selective Learning Network for Intelligent Diagnosis of Bimodal Ultrasound Video](https://mlsanity.com/paper/2409.01557/) (2024)
- [MTFL: Multi-Timescale Feature Learning for Weakly-Supervised Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/2410.05900/) (2024)
- [Advancing Video Anomaly Detection: A Bi-Directional Hybrid Framework for Enhanced Single- and Multi-Task Approaches](https://mlsanity.com/paper/2504.14753/) (2025)

## Abstract (arXiv)

Detecting abnormal activities in real-world surveillance videos is an important yet challenging task as the prior knowledge about video anomalies is usually limited or unavailable. Despite that many approaches have been developed to resolve this problem, few of them can capture the normal spatio-temporal patterns effectively and efficiently. Moreover, existing works seldom explicitly consider the local consistency at frame level and global coherence of temporal dynamics in video sequences. To this end, we propose Convolutional Transformer based Dual Discriminator Generative Adversarial Networks (CT-D2GAN) to perform unsupervised video anomaly detection. Specifically, we first present a convolutional transformer to perform future frame prediction. It contains three key components, i.e., a convolutional encoder to capture the spatial information of the input video clips, a temporal self-attention module to encode the temporal dynamics, and a convolutional decoder to integrate spatio-temporal features and predict the future frame. Next, a dual discriminator based adversarial training procedure, which jointly considers an image discriminator that can maintain the local consistency at frame-level and a video discriminator that can enforce the global coherence of temporal dynamics, is employed to enhance the future frame prediction. Finally, the prediction error is used to identify abnormal video frames. Thoroughly empirical studies on three public video anomaly detection datasets, i.e., UCSD Ped2, CUHK Avenue, and Shanghai Tech Campus, demonstrate the effectiveness of the proposed adversarial spatio-temporal modeling framework.