# Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video

arXiv: https://arxiv.org/abs/2108.03825 · HTML: https://mlsanity.com/paper/2108.03825/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [Learning Deep Representations of Appearance and Motion for Anomalous Event Detection](https://mlsanity.com/paper/1510.01553/) (2015)
- [Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition](https://mlsanity.com/paper/1907.13369/) (2019)
- [Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video](https://mlsanity.com/paper/1906.02549/) (2019)
- [Motion-Aware Feature for Improved Video Anomaly Detection](https://mlsanity.com/paper/1907.10211/) (2019)
- [Deep Learning for Detecting Multiple Space-Time Action Tubes in Videos](https://mlsanity.com/paper/1608.01529/) (2016)
- [Spatio-temporal Person Retrieval via Natural Language Queries](https://mlsanity.com/paper/1704.07945/) (2017)
- [Reinforcement Learning for Weakly Supervised Temporal Grounding of Natural Language in Untrimmed Videos](https://mlsanity.com/paper/2009.08614/) (2020)
- [Guess Where? Actor-Supervision for Spatiotemporal Action Localization](https://mlsanity.com/paper/1804.01824/) (2018)
- [Good Practices and A Strong Baseline for Traffic Anomaly Detection](https://mlsanity.com/paper/2105.03827/) (2021)
- [Spatio-Temporal Instance Learning: Action Tubes from Class Supervision](https://mlsanity.com/paper/1807.02800/) (2018)

## Built on by (most cited first, 30 of 32)

- [VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2308.11681/) (2023)
- [Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2302.05160/) (2023)
- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts](https://mlsanity.com/paper/2408.05905/) (2024)
- [Hierarchical Semantic Contrast for Scene-aware Video Anomaly Detection](https://mlsanity.com/paper/2303.13051/) (2023)
- [Vision-Based Traffic Accident Detection and Anticipation: A Survey](https://mlsanity.com/paper/2308.15985/) (2023)
- [Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2112.08996/) (2021)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Model](https://mlsanity.com/paper/2307.12545/) (2023)
- [Weakly Supervised Anomaly Detection: A Survey](https://mlsanity.com/paper/2302.04549/) (2023)
- [NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition](https://mlsanity.com/paper/2207.10388/) (2022)
- [Weakly-Supervised Video Anomaly Detection with Snippet Anomalous Attention](https://mlsanity.com/paper/2309.16309/) (2023)
- [CRASH: Crash Recognition and Anticipation System Harnessing with Context-Aware and Temporal Focus Attentions](https://mlsanity.com/paper/2407.17757/) (2024)
- [A Survey of Visual Sensory Anomaly Detection](https://mlsanity.com/paper/2202.07006/) (2022)
- [Consistency-based Self-supervised Learning for Temporal Anomaly Localization](https://mlsanity.com/paper/2208.05251/) (2022)
- [Long-Short Temporal Co-Teaching for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2303.18044/) (2023)
- [Temporal Saliency Query Network for Efficient Video Recognition](https://mlsanity.com/paper/2207.10379/) (2022)
- [Aligning Effective Tokens with Video Anomaly in Large Language Models](https://mlsanity.com/paper/2508.06350/) (2025)
- [Overlooked Video Classification in Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2210.06688/) (2022)
- [SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Model](https://mlsanity.com/paper/2504.10320/) (2025)
- [SAVCHOI: Detecting Suspicious Activities using Dense Video Captioning with Human Object Interactions](https://mlsanity.com/paper/2207.11838/) (2022)
- [Detection of Fights in Videos: A Comparison Study of Anomaly Detection and Action Recognition](https://mlsanity.com/paper/2205.11394/) (2022)
- [Towards Scalable Video Anomaly Retrieval: A Synthetic Video-Text Benchmark](https://mlsanity.com/paper/2506.01466/) (2025)
- [Towards Learning in Grey Spatiotemporal Systems: A Prophet to Non-consecutive Spatiotemporal Dynamics](https://mlsanity.com/paper/2208.08878/) (2022)
- [uTRAND: Unsupervised Anomaly Detection in Traffic Trajectories](https://mlsanity.com/paper/2404.12712/) (2024)
- [TeG: Temporal-Granularity Method for Anomaly Detection with Attention in Smart City Surveillance](https://mlsanity.com/paper/2411.11003/) (2024)
- [Uncertainty-Weighted Fusion of Image and Synthetic Event for Video Anomaly Detection](https://mlsanity.com/paper/2505.02393/) (2025)
- [Weakly Supervised Detection of Baby Cry](https://mlsanity.com/paper/2304.10001/) (2023)
- [ROAR: Robust Accident Recognition and Anticipation for Autonomous Driving](https://mlsanity.com/paper/2511.06226/) (2025)
- [Learning Where and When: Patch-Based Spatiotemporal Localization in Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2606.29498/) (2026)

## Abstract (arXiv)

In this paper, we introduce a novel task, referred to as Weakly-Supervised Spatio-Temporal Anomaly Detection (WSSTAD) in surveillance video. Specifically, given an untrimmed video, WSSTAD aims to localize a spatio-temporal tube (i.e., a sequence of bounding boxes at consecutive times) that encloses the abnormal event, with only coarse video-level annotations as supervision during training. To address this challenging task, we propose a dual-branch network which takes as input the proposals with multi-granularities in both spatial-temporal domains. Each branch employs a relationship reasoning module to capture the correlation between tubes/videolets, which can provide rich contextual information and complex entity relationships for the concept learning of abnormal behaviors. Mutually-guided Progressive Refinement framework is set up to employ dual-path mutual guidance in a recurrent manner, iteratively sharing auxiliary supervision information across branches. It impels the learned concepts of each branch to serve as a guide for its counterpart, which progressively refines the corresponding branch and the whole framework. Furthermore, we contribute two datasets, i.e., ST-UCF-Crime and STRA, consisting of videos containing spatio-temporal abnormal annotations to serve as the benchmarks for WSSTAD. We conduct extensive qualitative and quantitative evaluations to demonstrate the effectiveness of the proposed approach and analyze the key factors that contribute more to handle this task.