# Delving into CLIP latent space for Video Anomaly Recognition

arXiv: https://arxiv.org/abs/2310.02835 · HTML: https://mlsanity.com/paper/2310.02835/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Towards Total Recall in Industrial Anomaly Detection](https://mlsanity.com/paper/2106.08265/) (2021)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection](https://mlsanity.com/paper/2104.01633/) (2021)
- [A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction](https://mlsanity.com/paper/2108.06852/) (2021)
- [Generative Cooperative Learning for Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2203.03962/) (2022)
- [MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2211.15098/) (2022)
- [CLAWS: Clustering Assisted Weakly Supervised Learning with Normalcy Suppression for Anomalous Event Detection](https://mlsanity.com/paper/2011.12077/) (2020)
- [Learning Normal Dynamics in Videos with Meta Prototype Network](https://mlsanity.com/paper/2104.06689/) (2021)
- [GODS: Generalized One-class Discriminative Subspaces for Anomaly Detection](https://mlsanity.com/paper/1908.05884/) (2019)
- [Subspace Support Vector Data Description](https://mlsanity.com/paper/1802.03989/) (2018)
- [Convolutional Transformer based Dual Discriminator Generative Adversarial Networks for Video Anomaly Detection](https://mlsanity.com/paper/2107.13720/) (2021)
- [A Survey on Deep Learning Techniques for Video Anomaly Detection](https://mlsanity.com/paper/2009.14146/) (2020)

## Built on by (most cited first, 27 of 27)

- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2404.08531/) (2024)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [Video Anomaly Detection in 10 Years: A Survey and Outlook](https://mlsanity.com/paper/2405.19387/) (2024)
- [A Survey on Safe Multi-Modal Learning System](https://mlsanity.com/paper/2402.05355/) (2024)
- [MissionGNN: Hierarchical Multimodal GNN-based Weakly Supervised Video Anomaly Recognition with Mission-Specific Knowledge Graph Generation](https://mlsanity.com/paper/2406.18815/) (2024)
- [AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection](https://mlsanity.com/paper/2504.04495/) (2025)
- [Vision-Language Models Assisted Unsupervised Video Anomaly Detection](https://mlsanity.com/paper/2409.14109/) (2024)
- [Multimodal Attack Detection for Action Recognition Models](https://mlsanity.com/paper/2404.10790/) (2024)
- [PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation](https://mlsanity.com/paper/2410.22623/) (2024)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models](https://mlsanity.com/paper/2511.07299/) (2025)
- [Order Matters: On Parameter-Efficient Image-to-Video Probing for Recognizing Nearly Symmetric Actions](https://mlsanity.com/paper/2503.24298/) (2025)
- [Evaluation of Vision-LLMs in Surveillance Video](https://mlsanity.com/paper/2510.23190/) (2025)
- [Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models](https://mlsanity.com/paper/2506.02615/) (2025)
- [Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild](https://mlsanity.com/paper/2603.04727/) (2026)
- [Rethinking Open-World Video Anomaly Detection: Diagnosing Definition Blindness](https://mlsanity.com/paper/2607.20780/) (2026)
- [Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning](https://mlsanity.com/paper/2411.09072/) (2024)
- [Can Text-to-Video Generation help Video-Language Alignment?](https://mlsanity.com/paper/2503.18507/) (2025)
- [MReg: A Novel Regression Model with MoE-based Video Feature Mining for Mitral Regurgitation Diagnosis](https://mlsanity.com/paper/2506.23648/) (2025)
- [Benchmarking Compact VLMs for Clip-Level Surveillance Anomaly Detection Under Weak Supervision](https://mlsanity.com/paper/2603.13306/) (2026)
- [MuST-VAD: Mutual Structured Learning for Video Anomaly Detection](https://mlsanity.com/paper/2608.06913/) (2026)
- [LMM-Regularized CLIP Embeddings for Image Classification](https://mlsanity.com/paper/2412.11663/) (2024)
- [Detection, Retrieval, and Explanation Unified: A Violence Detection System Based on Knowledge Graphs and GAT](https://mlsanity.com/paper/2501.06224/) (2025)
- [MissionHD: Hyperdimensional Refinement of Distribution-Deficient Reasoning Graphs for Video Anomaly Detection](https://mlsanity.com/paper/2508.14746/) (2025)
- [TorchTraceAP: A New Benchmark Dataset for Detecting Performance Anti-Patterns in Computer Vision Models](https://mlsanity.com/paper/2512.14141/) (2025)
- [FedVAR: Prototype-Aligned Federated Framework for Video Anomaly Recognition](https://mlsanity.com/paper/2608.06876/) (2026)

## Abstract (arXiv)

We tackle the complex problem of detecting and recognising anomalies in surveillance videos at the frame level, utilising only video-level supervision. We introduce the novel method AnomalyCLIP, the first to combine Large Language and Vision (LLV) models, such as CLIP, with multiple instance learning for joint video anomaly detection and classification. Our approach specifically involves manipulating the latent CLIP feature space to identify the normal event subspace, which in turn allows us to effectively learn text-driven directions for abnormal events. When anomalous frames are projected onto these directions, they exhibit a large feature magnitude if they belong to a particular class. We also introduce a computationally efficient Transformer architecture to model short- and long-term temporal dependencies between frames, ultimately producing the final anomaly score and class prediction probabilities. We compare AnomalyCLIP against state-of-the-art methods considering three major anomaly detection benchmarks, i.e. ShanghaiTech, UCF-Crime, and XD-Violence, and empirically show that it outperforms baselines in recognising video anomalies.