# VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs

arXiv: https://arxiv.org/abs/2406.10326 · HTML: https://mlsanity.com/paper/2406.10326/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 26 of 26)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens](https://mlsanity.com/paper/2404.03413/) (2024)
- [VideoLCM: Video Latent Consistency Model](https://mlsanity.com/paper/2312.09109/) (2023)
- [Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models](https://mlsanity.com/paper/2302.05087/) (2023)
- [Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly](https://mlsanity.com/paper/2405.00181/) (2024)
- [How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs](https://mlsanity.com/paper/2405.03690/) (2024)

## Built on by (most cited first, 20 of 20)

- [Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection](https://mlsanity.com/paper/2506.00979/) (2025)
- [Survey on AI-Generated Media Detection: From Non-MLLM to MLLM](https://mlsanity.com/paper/2502.05240/) (2025)
- [AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding](https://mlsanity.com/paper/2507.02591/) (2025)
- [VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding](https://mlsanity.com/paper/2507.21507/) (2025)
- [AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond](https://mlsanity.com/paper/2509.26636/) (2025)
- [Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs](https://mlsanity.com/paper/2510.09201/) (2025)
- [A2Seek: Towards Reasoning-Centric Benchmark for Aerial Anomaly Understanding](https://mlsanity.com/paper/2505.21962/) (2025)
- [SmartHome-Bench: A Comprehensive Benchmark for Video Anomaly Detection in Smart Homes Using Multi-Modal Large Language Models](https://mlsanity.com/paper/2506.12992/) (2025)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [Detecting Contextual Anomalies by Discovering Consistent Spatial Regions](https://mlsanity.com/paper/2501.08470/) (2025)
- [PSyDUCK: Training-Free Steganography for Latent Diffusion](https://mlsanity.com/paper/2501.19172/) (2025)
- [Text-guided Fine-Grained Video Anomaly Understanding](https://mlsanity.com/paper/2511.00524/) (2025)
- [ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantification](https://mlsanity.com/paper/2512.03101/) (2025)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding](https://mlsanity.com/paper/2604.12335/) (2026)
- [Advancing Reliable Synthetic Video Detection: Insights from the SAFE Challenge](https://mlsanity.com/paper/2605.06912/) (2026)
- [LADBench: A Benchmark for Logical Fault Detection in Images](https://mlsanity.com/paper/2606.17433/) (2026)
- [Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning](https://mlsanity.com/paper/2608.11260/) (2026)
- [What's the Catch? Evaluating Temporal Consistency in Vision-Language Models](https://mlsanity.com/paper/2608.23474/) (2026)
- [Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding](https://mlsanity.com/paper/2610.01754/) (2026)

## Abstract (arXiv)

The recent developments in Large Multi-modal Video Models (Video-LMMs) have significantly enhanced our ability to interpret and analyze video data. Despite their impressive capabilities, current Video-LMMs have not been evaluated for anomaly detection tasks, which is critical to their deployment in practical scenarios e.g., towards identifying deepfakes, manipulated video content, traffic accidents and crimes. In this paper, we introduce VANE-Bench, a benchmark designed to assess the proficiency of Video-LMMs in detecting and localizing anomalies and inconsistencies in videos. Our dataset comprises an array of videos synthetically generated using existing state-of-the-art text-to-video generation models, encompassing a variety of subtle anomalies and inconsistencies grouped into five categories: unnatural transformations, unnatural appearance, pass-through, disappearance and sudden appearance. Additionally, our benchmark features real-world samples from existing anomaly detection datasets, focusing on crime-related irregularities, atypical pedestrian behavior, and unusual events. The task is structured as a visual question-answering challenge to gauge the models' ability to accurately detect and localize the anomalies within the videos. We evaluate nine existing Video-LMMs, both open and closed sources, on this benchmarking task and find that most of the models encounter difficulties in effectively identifying the subtle anomalies. In conclusion, our research offers significant insights into the current capabilities of Video-LMMs in the realm of anomaly detection, highlighting the importance of our work in evaluating and improving these models for real-world applications. Our code and data is available at https://hananshafi.github.io/vane-benchmark/