# SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection

arXiv: https://arxiv.org/abs/2207.08003 · HTML: https://mlsanity.com/paper/2207.08003/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 51 of its 96 references that are in the map.
- First posted on arXiv 16 Jul 2022.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://mlsanity.com/paper/2011.07491/) (2020)
- [MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection](https://mlsanity.com/paper/2104.01633/) (2021)
- [A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction](https://mlsanity.com/paper/2108.06852/) (2021)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection](https://mlsanity.com/paper/2111.09099/) (2021)
- [Anomaly Detection in Video Sequence with Appearance-Motion Correspondence](https://mlsanity.com/paper/1908.06351/) (2019)
- [Object-centric Auto-encoders and Dummy Anomalies for Abnormal Event Detection in Video](https://mlsanity.com/paper/1812.04960/) (2018)
- [Generative Cooperative Learning for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2203.03962/) (2022)
- [UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection](https://mlsanity.com/paper/2111.08644/) (2021)
- [Street Scene: A new dataset and evaluation protocol for video anomaly detection](https://mlsanity.com/paper/1902.05872/) (2019)
- [A Survey of Single-Scene Video Anomaly Detection](https://mlsanity.com/paper/2004.05993/) (2020)

## Built on by (most cited first, 30 of 45)

- [Normalizing Flows for Human Pose Anomaly Detection](https://mlsanity.com/paper/2211.10946/) (2022)
- [Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection](https://mlsanity.com/paper/2307.07205/) (2023)
- [Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors](https://mlsanity.com/paper/2306.12041/) (2023)
- [Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection](https://mlsanity.com/paper/2209.12148/) (2022)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection](https://mlsanity.com/paper/2403.14497/) (2024)
- [Cross-Domain Video Anomaly Detection without Target Domain Adaptation](https://mlsanity.com/paper/2212.07010/) (2022)
- [Context Recovery and Knowledge Retrieval: A Novel Two-Stream Framework for Video Anomaly Detection](https://mlsanity.com/paper/2209.02899/) (2022)
- [Video Anomaly Detection in 10 Years: A Survey and Outlook](https://mlsanity.com/paper/2405.19387/) (2024)
- [Contracting Skeletal Kinematics for Human-Related Video Anomaly Detection](https://mlsanity.com/paper/2301.09489/) (2023)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection](https://mlsanity.com/paper/2403.12172/) (2024)
- [ComplexVAD: Detecting Interaction Anomalies in Video](https://mlsanity.com/paper/2501.09733/) (2025)
- [Foundation Models and Transformers for Anomaly Detection: A Survey](https://mlsanity.com/paper/2507.15905/) (2025)
- [Evaluating the Effectiveness of Video Anomaly Detection in the Wild: Online Learning and Inference for Real-world Deployment](https://mlsanity.com/paper/2404.18747/) (2024)
- [DA-Flow: Dual Attention Normalizing Flow for Skeleton-based Video Anomaly Detection](https://mlsanity.com/paper/2406.02976/) (2024)
- [Sequential keypoint density estimator: an overlooked baseline of skeleton-based video anomaly detection](https://mlsanity.com/paper/2506.18368/) (2025)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [Dynamic Distinction Learning: Adaptive Pseudo Anomalies for Video Anomaly Detection](https://mlsanity.com/paper/2404.04986/) (2024)
- [Towards Adaptive Human-centric Video Anomaly Detection: A Comprehensive Framework and A New Benchmark](https://mlsanity.com/paper/2408.14329/) (2024)
- [Human-Centric Video Anomaly Detection Through Spatio-Temporal Pose Tokenization and Transformer](https://mlsanity.com/paper/2408.15185/) (2024)
- [Interleaving One-Class and Weakly-Supervised Models with Adaptive Thresholding for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2401.13551/) (2024)
- [Bidirectional skip-frame prediction for video anomaly detection with intra-domain disparity-driven attention](https://mlsanity.com/paper/2407.15424/) (2024)
- [Vision Technologies with Applications in Traffic Surveillance Systems: A Holistic Survey](https://mlsanity.com/paper/2412.00348/) (2024)
- [Autoregressive Denoising Score Matching is a Good Video Anomaly Detector](https://mlsanity.com/paper/2506.23282/) (2025)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [A Survey on Video Anomaly Detection via Deep Learning: Human, Vehicle, and Environment](https://mlsanity.com/paper/2508.14203/) (2025)
- [Leveraging Multimodal LLM Descriptions of Activity for Explainable Semi-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2510.14896/) (2025)
- [Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation](https://mlsanity.com/paper/2211.15597/) (2022)
- [IPAD: Industrial Process Anomaly Detection Dataset](https://mlsanity.com/paper/2404.15033/) (2024)

## Abstract (arXiv)

A self-supervised multi-task learning (SSMTL) framework for video anomaly detection was recently introduced in literature. Due to its highly accurate results, the method attracted the attention of many researchers. In this work, we revisit the self-supervised multi-task learning framework, proposing several updates to the original method. First, we study various detection methods, e.g. based on detecting high-motion regions using optical flow or background subtraction, since we believe the currently used pre-trained YOLOv3 is suboptimal, e.g. objects in motion or objects from unknown classes are never detected. Second, we modernize the 3D convolutional backbone by introducing multi-head self-attention modules, inspired by the recent success of vision transformers. As such, we alternatively introduce both 2D and 3D convolutional vision transformer (CvT) blocks. Third, in our attempt to further improve the model, we study additional self-supervised learning tasks, such as predicting segmentation maps through knowledge distillation, solving jigsaw puzzles, estimating body pose through knowledge distillation, predicting masked regions (inpainting), and adversarial learning with pseudo-anomalies. We conduct experiments to assess the performance impact of the introduced changes. Upon finding more promising configurations of the framework, dubbed SSMTL++v1 and SSMTL++v2, we extend our preliminary experiments to more data sets, demonstrating that our performance gains are consistent across all data sets. In most cases, our results on Avenue, ShanghaiTech and UBnormal raise the state-of-the-art performance bar to a new level.