mlsanityData to 7 Oct 2026

Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM

arXiv 2406.12235 ↗First posted 18 Jun 2024cs.CVcited by 63

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Towards open-ended Video Anomaly Detection (VAD), existing methods often exhibit biased detection when faced with challenging or unseen events and lack interpretability. To address these drawbacks, we propose Holmes-VAD, a novel framework that leverages precise temporal supervision and rich multimodal instructions to enable accurate anomaly localization and comprehensive explanations. Firstly, towards unbiased and explainable VAD system, we construct the first large-scale multimodal VAD instruction-tuning benchmark, i.e., VAD-Instruct50k. This dataset is created using a carefully designed semi-automatic labeling paradigm. Efficient single-frame annotations are applied to the collected untrimmed videos, which are then synthesized into high-quality analyses of both abnormal and normal video clips using a robust off-the-shelf video captioner and a large language model (LLM). Building upon the VAD-Instruct50k dataset, we develop a customized solution for interpretable video anomaly detection. We train a lightweight temporal sampler to select frames with high anomaly response and fine-tune a multimodal large language model (LLM) to generate explanatory content. Extensive experimental results validate the generality and interpretability of the proposed Holmes-VAD, establishing it as a novel interpretable technique for real-world video anomaly analysis. To support the community, our benchmark and model will be publicly available at https://holmesvad.github.io.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 71 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

57 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models29 Aug
  2. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection22 Aug

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2023Mistral 7Bcited by 3,065
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
38 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
  9. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 560
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  11. 2017Future Frame Prediction for Anomaly Detection -- A New Baselinecited by 391
  12. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  13. 2016Learning Temporal Regularity in Video Sequencescited by 254
  14. 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 235
  15. 2021Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learningcited by 226
  16. 2019Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detectioncited by 171
  17. 2021MIST: Multiple Instance Self-Training Framework for Video Anomaly Detectioncited by 137
  18. 2021A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Predictioncited by 129
  19. 2022Generative Cooperative Learning for Unsupervised Video Anomaly Detectioncited by 102
  20. 2023Dual Memory Units with Uncertainty Regulation for Weakly Supervised Video Anomaly Detectioncited by 88
  21. 2022MGFN: Magnitude-Contrastive Glance-and-Focus Network for Weakly-Supervised Video Anomaly Detectioncited by 87
  22. 2024Harnessing Large Language Models for Training-free Video Anomaly Detectioncited by 85
  23. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  24. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  25. 2019GODS: Generalized One-class Discriminative Subspaces for Anomaly Detectioncited by 64
  26. 2023Unbiased Multiple Instance Learning for Weakly Supervised Video Anomaly Detectioncited by 64
  27. 2020SF-Net: Single-Frame Supervision for Temporal Action Localizationcited by 62
  28. 2023Open-Vocabulary Video Anomaly Detectioncited by 57
  29. 2023Video Event Restoration Based on Keyframes for Video Anomaly Detectioncited by 54
  30. 2024Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detectioncited by 49
  31. 2024Video Anomaly Detection and Explanation via Large Language Modelscited by 45
  32. 2023Learning Prompt-Enhanced Context Features for Weakly-Supervised Video Anomaly Detectioncited by 43
  33. 2021Learning Action Completeness from Points for Weakly-supervised Temporal Action Localizationcited by 42
  34. 2023CLIP-guided Prototype Modulating for Few-shot Action Recognitioncited by 41
  35. 2022DyAnNet: A Scene Dynamicity Guided Self-Trained Video Anomaly Detection Networkcited by 26
  36. 2023Exploring Diffusion Models for Unsupervised Video Anomaly Detectioncited by 23
  37. 2022Video Moment Retrieval from Text Queries via Single Frame Annotationcited by 19
  38. 2019Anomaly Locality in Video Surveillancecited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 31; 2026: 24.

63 papers.

Sort:

By area:

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  3. 2024Deep Learning for Video Anomaly Detection: A Reviewcited by 27
  4. 2024Large Language Models for Anomaly and Out-of-Distribution Detection: A Surveycited by 24
  5. 2024VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Modelscited by 24
  6. 2025VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuningcited by 24
  7. 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thoughtcited by 23
  8. 2025Deep Learning Advancements in Anomaly Detection: A Comprehensive Surveycited by 15
  9. 2025PANDA: Towards Generalist Video Anomaly Detection via Agentic AI Engineercited by 15
  10. 2025VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Treecited by 15
  11. 2024Do Language Models Understand Time?cited by 14
  12. 2025Flashback: Memory-Driven Zero-shot, Real-time Video Anomaly Detectioncited by 12
38 more
  1. 2025Sherlock: Towards Multi-scene Video Abnormal Event Extraction and Localization via a Global-local Spatial-sensitive LLMcited by 10
  2. 2025Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modelingcited by 10
  3. 2024Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomalycited by 9
  4. 2025Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detectioncited by 9
  5. 2025A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysiscited by 9
  6. 2024Quo Vadis, Anomaly Detection? LLMs and VLMs in the Spotlightcited by 8
  7. 2025Language-guided Open-world Video Anomaly Detection under Weak Supervisioncited by 8
  8. 2025HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMscited by 8
  9. 2025Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Promptingcited by 8
  10. 2025VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understandingcited by 6
  11. 2025SmartHome-Bench: A Comprehensive Benchmark for Video Anomaly Detection in Smart Homes Using Multi-Modal Large Language Modelscited by 4
  12. 2026Instance-Aligned Captions for Explainable Video Anomaly Detectioncited by 4
  13. 2025Detecting Contextual Anomalies by Discovering Consistent Spatial Regionscited by 3
  14. 2025Foundation Models for Anomaly Detection: Vision and Challengescited by 3
  15. 2025Large Language Models for Crash Detection in Video: A Survey of Methods, Datasets, and Challengescited by 3
  16. 2025Evaluation of Vision-LLMs in Surveillance Videocited by 3
  17. 2026TAU-R1: Visual Language Model for Traffic Anomaly Understandingcited by 3
  18. 2025DUAL-VAD: Dual Benchmarks and Anomaly-Focused Sampling for Video Anomaly Detectioncited by 2
  19. 2025ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantificationcited by 2
  20. 2025HeadHunt-VAD: Hunting Robust Anomaly-Sensitive Heads in MLLM for Tuning-Free Video Anomaly Detectioncited by 2
  21. 2026Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detectioncited by 2
  22. 2026TD-VAD: Breaking Visual Dependence in Video Anomaly Detection with Text-Driven Learningcited by 2
  23. 2025Omni-SILA: Towards Omni-scene Driven Visual Sentiment Identifying, Locating and Attributing in Videoscited by 1
  24. 2025Hoi2Threat: An Interpretable Threat Detection Method for Human Violence Scenarios Guided by Human-Object Interactioncited by 1
  25. 2025TRACES: Temporal Recall with Contextual Embeddings for Real-Time Video Anomaly Detectioncited by 1
  26. 2025CADE: Continual Weakly-supervised Video Anomaly Detection with Ensemblescited by 1
  27. 2026PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understandingcited by 1
  28. 2026Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionalscited by 1
  29. 2026DeepSVU: Towards In-depth Security-oriented Video Understanding via Unified Physical-world Regularized MoEcited by 1
  30. 2026Reasoning-Guided Grounding: Elevating Video Anomaly Detection through Multimodal Large Language Modelscited by 1
  31. 2026COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detectioncited by 1
  32. 2026From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoningcited by 1
  33. 2025Explaining Low Perception Model Competency with High-Competency Counterfactualscited by 0
  34. 2025Runtime Failure Hunting for Physics Engine Based Software Systems: How Far Can We Go?cited by 0
  35. 2025MA-CBP: A Criminal Behavior Prediction Framework Based on Multi-Agent Asynchronous Collaborationcited by 0
  36. 2025Live-E2T: Real-time Threat Monitoring in Video via Deduplicated Event Reasoning and Chain-of-Thoughtcited by 0
  37. 2025GMFVAD: Using Grained Multi-modal Feature to Improve Video Anomaly Detectioncited by 0
  38. 2026Advancing Adaptive Multi-Stage Video Anomaly Reasoning: A Benchmark Dataset and Methodcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity
  2. 2025Leveraging Multimodal LLM Descriptions of Activity for Explainable Semi-Supervised Video Anomaly Detection
  3. 2025EventVAD: Training-Free Event-Aware Video Anomaly Detection
  4. 2025SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Model
  5. 2026Is Video Anomaly Detection Misframed? Evidence from LLM-Based and Multi-Scene Models
  6. 2025The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM
  7. 2026Probe-VAD: Ordinal Likelihood Probing for Training-Free Video Anomaly Detection
  8. 2026Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild