# CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detection

arXiv: https://arxiv.org/abs/2212.05136 · HTML: https://mlsanity.com/paper/2212.05136/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 108 references that are in the map.
- First posted on arXiv 9 Dec 2022.

## Builds on (most cited first, 30 of 73)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)

## Built on by (most cited first, 30 of 67)

- [VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2308.11681/) (2023)
- [Harnessing Large Language Models for Training-free Video Anomaly Detection](https://mlsanity.com/paper/2404.01014/) (2024)
- [Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity](https://mlsanity.com/paper/2412.06171/) (2024)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [Open-Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2311.07042/) (2023)
- [Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2404.08531/) (2024)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models](https://mlsanity.com/paper/2412.01095/) (2024)
- [Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2410.10289/) (2024)
- [Video Anomaly Detection in 10 Years: A Survey and Outlook](https://mlsanity.com/paper/2405.19387/) (2024)
- [VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Tree](https://mlsanity.com/paper/2510.22693/) (2025)
- [Anomize: Better Open Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2503.18094/) (2025)
- [ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection](https://mlsanity.com/paper/2311.00729/) (2023)
- [Flashback: Memory-Driven Zero-shot, Real-time Video Anomaly Detection](https://mlsanity.com/paper/2505.15205/) (2025)
- [A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysis](https://mlsanity.com/paper/2511.00962/) (2025)
- [MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection](https://mlsanity.com/paper/2510.21449/) (2025)
- [Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detection](https://mlsanity.com/paper/2503.03562/) (2025)
- [Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly](https://mlsanity.com/paper/2412.07183/) (2024)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [Language-guided Open-world Video Anomaly Detection under Weak Supervision](https://mlsanity.com/paper/2503.13160/) (2025)
- [Foundation Models and Transformers for Anomaly Detection: A Survey](https://mlsanity.com/paper/2507.15905/) (2025)
- [VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding](https://mlsanity.com/paper/2507.21507/) (2025)
- [Aligning Effective Tokens with Video Anomaly in Large Language Models](https://mlsanity.com/paper/2508.06350/) (2025)
- [Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting](https://mlsanity.com/paper/2510.02155/) (2025)
- [No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection](https://mlsanity.com/paper/2602.19248/) (2026)
- [Anomaly Detection for People with Visual Impairments Using an Egocentric 360-Degree Camera](https://mlsanity.com/paper/2411.10945/) (2024)
- [AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection](https://mlsanity.com/paper/2504.04495/) (2025)
- [Cross-Domain Learning for Video Anomaly Detection with Limited Supervision](https://mlsanity.com/paper/2408.05191/) (2024)
- [Mixture of Experts Guided by Gaussian Splatters Matters: A new Approach to Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2508.06318/) (2025)
- [Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks](https://mlsanity.com/paper/2511.19474/) (2025)

## Abstract (arXiv)

Video anomaly detection (VAD) -- commonly formulated as a multiple-instance learning problem in a weakly-supervised manner due to its labor-intensive nature -- is a challenging problem in video surveillance where the frames of anomaly need to be localized in an untrimmed video. In this paper, we first propose to utilize the ViT-encoded visual features from CLIP, in contrast with the conventional C3D or I3D features in the domain, to efficiently extract discriminative representations in the novel technique. We then model temporal dependencies and nominate the snippets of interest by leveraging our proposed Temporal Self-Attention (TSA). The ablation study confirms the effectiveness of TSA and ViT feature. The extensive experiments show that our proposed CLIP-TSA outperforms the existing state-of-the-art (SOTA) methods by a large margin on three commonly-used benchmark datasets in the VAD problem (UCF-Crime, ShanghaiTech Campus, and XD-Violence). Our source code is available at https://github.com/joos2010kj/CLIP-TSA.