# Open-Vocabulary Video Anomaly Detection

arXiv: https://arxiv.org/abs/2311.07042 · HTML: https://mlsanity.com/paper/2311.07042/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 66 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://mlsanity.com/paper/2011.07491/) (2020)
- [A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction](https://mlsanity.com/paper/2108.06852/) (2021)
- [VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2308.11681/) (2023)
- [Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detection](https://mlsanity.com/paper/2203.14506/) (2022)

## Built on by (most cited first, 30 of 57)

- [Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts](https://mlsanity.com/paper/2403.06495/) (2024)
- [Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity](https://mlsanity.com/paper/2412.06171/) (2024)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [Hawk: Learning to Understand Open-World Video Anomalies](https://mlsanity.com/paper/2405.16886/) (2024)
- [Deep Graph Anomaly Detection: A Survey and New Perspectives](https://mlsanity.com/paper/2409.09957/) (2024)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models](https://mlsanity.com/paper/2412.01095/) (2024)
- [VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.23504/) (2025)
- [Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2410.10289/) (2024)
- [SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Models](https://mlsanity.com/paper/2505.12589/) (2025)
- [Anomize: Better Open Vocabulary Video Anomaly Detection](https://mlsanity.com/paper/2503.18094/) (2025)
- [Flashback: Memory-Driven Zero-shot, Real-time Video Anomaly Detection](https://mlsanity.com/paper/2505.15205/) (2025)
- [A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysis](https://mlsanity.com/paper/2511.00962/) (2025)
- [Privacy-Preserving Video Anomaly Detection: A Survey](https://mlsanity.com/paper/2411.14565/) (2024)
- [Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detection](https://mlsanity.com/paper/2503.03562/) (2025)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection](https://mlsanity.com/paper/2411.10922/) (2024)
- [Language-guided Open-world Video Anomaly Detection under Weak Supervision](https://mlsanity.com/paper/2503.13160/) (2025)
- [Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting](https://mlsanity.com/paper/2510.02155/) (2025)
- [No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection](https://mlsanity.com/paper/2602.19248/) (2026)
- [AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection](https://mlsanity.com/paper/2504.04495/) (2025)
- [Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection](https://mlsanity.com/paper/2412.03044/) (2024)
- [Zero-Shot Out-of-Distribution Detection with Outlier Label Exposure](https://mlsanity.com/paper/2406.01170/) (2024)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [Leveraging Multimodal LLM Descriptions of Activity for Explainable Semi-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2510.14896/) (2025)
- [Instance-Aligned Captions for Explainable Video Anomaly Detection](https://mlsanity.com/paper/2601.08155/) (2026)
- [Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection](https://mlsanity.com/paper/2602.24021/) (2026)
- [Rethinking Metrics and Benchmarks of Video Anomaly Detection](https://mlsanity.com/paper/2505.19022/) (2025)
- [Learning Event Completeness for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2506.13095/) (2025)

## Abstract (arXiv)

Video anomaly detection (VAD) with weak supervision has achieved remarkable performance in utilizing video-level labels to discriminate whether a video frame is normal or abnormal. However, current approaches are inherently limited to a closed-set setting and may struggle in open-world applications where there can be anomaly categories in the test data unseen during training. A few recent studies attempt to tackle a more realistic setting, open-set VAD, which aims to detect unseen anomalies given seen anomalies and normal videos. However, such a setting focuses on predicting frame anomaly scores, having no ability to recognize the specific categories of anomalies, despite the fact that this ability is essential for building more informed video surveillance systems. This paper takes a step further and explores open-vocabulary video anomaly detection (OVVAD), in which we aim to leverage pre-trained large models to detect and categorize seen and unseen anomalies. To this end, we propose a model that decouples OVVAD into two mutually complementary tasks -- class-agnostic detection and class-specific classification -- and jointly optimizes both tasks. Particularly, we devise a semantic knowledge injection module to introduce semantic knowledge from large language models for the detection task, and design a novel anomaly synthesis module to generate pseudo unseen anomaly videos with the help of large vision generation models for the classification task. These semantic knowledge and synthesis anomalies substantially extend our model's capability in detecting and categorizing a variety of seen and unseen anomalies. Extensive experiments on three widely-used benchmarks demonstrate our model achieves state-of-the-art performance on OVVAD task.