# Video Anomaly Detection and Explanation via Large Language Models

arXiv: https://arxiv.org/abs/2401.05702 · HTML: https://mlsanity.com/paper/2401.05702/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [An Empirical Study of Training End-to-End Vision-and-Language Transformers](https://mlsanity.com/paper/2111.02387/) (2021)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2302.05160/) (2023)
- [CLAWS: Clustering Assisted Weakly Supervised Learning with Normalcy Suppression for Anomalous Event Detection](https://mlsanity.com/paper/2011.12077/) (2020)
- [Unbiased Multiple Instance Learning for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2303.12369/) (2023)
- [Localizing Anomalies from Weakly-Labeled Videos](https://mlsanity.com/paper/2008.08944/) (2020)
- [Motion-Aware Feature for Improved Video Anomaly Detection](https://mlsanity.com/paper/1907.10211/) (2019)
- [Exploiting Completeness and Uncertainty of Pseudo Labels for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2212.04090/) (2022)
- [Global Information Guided Video Anomaly Detection](https://mlsanity.com/paper/2104.06813/) (2021)

## Built on by (most cited first, 30 of 47)

- [Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity](https://mlsanity.com/paper/2412.06171/) (2024)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2407.10299/) (2024)
- [Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2502.07601/) (2025)
- [Advancing Video Anomaly Detection: A Concise Review and a New Dataset](https://mlsanity.com/paper/2402.04857/) (2024)
- [Hawk: Learning to Understand Open-World Video Anomalies](https://mlsanity.com/paper/2405.16886/) (2024)
- [SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations](https://mlsanity.com/paper/2412.06878/) (2024)
- [Deep Learning for Video Anomaly Detection: A Review](https://mlsanity.com/paper/2409.05383/) (2024)
- [VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models](https://mlsanity.com/paper/2412.01095/) (2024)
- [Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought](https://mlsanity.com/paper/2505.19877/) (2025)
- [VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.23504/) (2025)
- [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024)
- [VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation](https://mlsanity.com/paper/2505.14640/) (2025)
- [VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Tree](https://mlsanity.com/paper/2510.22693/) (2025)
- [Reasoning Segmentation for Images and Videos: A Survey](https://mlsanity.com/paper/2505.18816/) (2025)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [Quo Vadis, Anomaly Detection? LLMs and VLMs in the Spotlight](https://mlsanity.com/paper/2412.18298/) (2024)
- [VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding](https://mlsanity.com/paper/2507.21507/) (2025)
- [Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting](https://mlsanity.com/paper/2510.02155/) (2025)
- [A2Seek: Towards Reasoning-Centric Benchmark for Aerial Anomaly Understanding](https://mlsanity.com/paper/2505.21962/) (2025)
- [Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems](https://mlsanity.com/paper/2409.02278/) (2024)
- [SmartHome-Bench: A Comprehensive Benchmark for Video Anomaly Detection in Smart Homes Using Multi-Modal Large Language Models](https://mlsanity.com/paper/2506.12992/) (2025)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [Leveraging Multimodal LLM Descriptions of Activity for Explainable Semi-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2510.14896/) (2025)
- [VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models](https://mlsanity.com/paper/2511.07299/) (2025)
- [Instance-Aligned Captions for Explainable Video Anomaly Detection](https://mlsanity.com/paper/2601.08155/) (2026)
- [Foundation Models for Anomaly Detection: Vision and Challenges](https://mlsanity.com/paper/2502.06911/) (2025)
- [TAU-R1: Visual Language Model for Traffic Anomaly Understanding](https://mlsanity.com/paper/2603.19098/) (2026)
- [INVARLLM: LLM-assisted Physical Invariant Extraction for Cyber-Physical Systems Anomaly Detection](https://mlsanity.com/paper/2411.10918/) (2024)
- [Sage Deer: A Super-Aligned Driving Generalist Is Your Copilot](https://mlsanity.com/paper/2505.10257/) (2025)

## Abstract (arXiv)

Video Anomaly Detection (VAD) aims to localize abnormal events on the timeline of long-range surveillance videos. Anomaly-scoring-based methods have been prevailing for years but suffer from the high complexity of thresholding and low explanability of detection results. In this paper, we conduct pioneer research on equipping video-based large language models (VLLMs) in the framework of VAD, making the VAD model free from thresholds and able to explain the reasons for the detected anomalies. We introduce a novel network module Long-Term Context (LTC) to mitigate the incapability of VLLMs in long-range context modeling. We design a three-phase training method to improve the efficiency of fine-tuning VLLMs by substantially minimizing the requirements for VAD data and lowering the costs of annotating instruction-tuning data. Our trained model achieves the top performance on the anomaly videos of the UCF-Crime and TAD benchmarks, with the AUC improvements of +3.86% and +4.96%, respectively. More impressively, our approach can provide textual explanations for detected anomalies.