# Sherlock: Towards Multi-scene Video Abnormal Event Extraction and Localization via a Global-local Spatial-sensitive LLM

arXiv: https://arxiv.org/abs/2502.18863 · HTML: https://mlsanity.com/paper/2502.18863/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 78 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 27 of 27)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Valley: Video Assistant with Large Language model Enhanced abilitY](https://mlsanity.com/paper/2306.07207/) (2023)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2308.11681/) (2023)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [RWF-2000: An Open Large Scale Video Database for Violence Detection](https://mlsanity.com/paper/1911.05913/) (2019)
- [Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2401.00901/) (2023)
- [DiaASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis](https://mlsanity.com/paper/2211.05705/) (2022)
- [PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN](https://mlsanity.com/paper/1708.01956/) (2017)
- [Anomaly Locality in Video Surveillance](https://mlsanity.com/paper/1901.10364/) (2019)
- [ManiFoundation Model for General-Purpose Robotic Manipulation of Contact Synthesis with Arbitrary Objects and Robots](https://mlsanity.com/paper/2405.06964/) (2024)

## Built on by (most cited first, 11 of 11)

- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [CueBench: Advancing Unified Understanding of Context-Aware Video Anomalies in Real-World](https://mlsanity.com/paper/2511.00613/) (2025)
- [SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere](https://mlsanity.com/paper/2605.08003/) (2026)
- [Cerberus: Real-Time Video Anomaly Detection via Cascaded Vision-Language Models](https://mlsanity.com/paper/2510.16290/) (2025)
- [Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals](https://mlsanity.com/paper/2602.08882/) (2026)
- [DeepSVU: Towards In-depth Security-oriented Video Understanding via Unified Physical-world Regularized MoE](https://mlsanity.com/paper/2602.18019/) (2026)
- [SRVAU-R1: Enhancing Video Anomaly Understanding via Reflection-Aware Learning](https://mlsanity.com/paper/2602.01004/) (2026)
- [From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models](https://mlsanity.com/paper/2606.26196/) (2026)
- [EvoSherlock: Towards Agentic Lifelong Evolution for Unseen Long-Tailed Security-Critical Events in Videos](https://mlsanity.com/paper/2609.19201/) (2026)
- [Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding](https://mlsanity.com/paper/2610.01754/) (2026)
- [Learning Temporal Evidence Selection for Video Anomaly Understanding](https://mlsanity.com/paper/2610.16299/) (2026)

## Abstract (arXiv)

Prior studies on Video Anomaly Detection (VAD) mainly focus on detecting whether each video frame is abnormal or not in the video, which largely ignore the structured video semantic information (i.e., what, when, and where does the abnormal event happen). With this in mind, we propose a new chat-paradigm Multi-scene Video Abnormal Event Extraction and Localization (M-VAE) task, aiming to extract the abnormal event quadruples (i.e., subject, event type, object, scene) and localize such event. Further, this paper believes that this new task faces two key challenges, i.e., global-local spatial modeling and global-local spatial balancing. To this end, this paper proposes a Global-local Spatial-sensitive Large Language Model (LLM) named Sherlock, i.e., acting like Sherlock Holmes to track down the criminal events, for this M-VAE task. Specifically, this model designs a Global-local Spatial-enhanced MoE (GSM) module and a Spatial Imbalance Regulator (SIR) to address the two challenges respectively. Extensive experiments on our M-VAE instruction dataset show the significant advantages of Sherlock over several advanced Video-LLMs. This justifies the importance of global-local spatial information for the M-VAE task and the effectiveness of Sherlock in capturing such information.