# HyCoVAD: A Hybrid SSL-LLM Model for Complex Video Anomaly Detection

arXiv: https://arxiv.org/abs/2509.22544 · HTML: https://mlsanity.com/paper/2509.22544/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 30 of 32)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [nnFormer: Interleaved Transformer for Volumetric Segmentation](https://mlsanity.com/paper/2109.03201/) (2021)
- [Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://mlsanity.com/paper/2011.07491/) (2020)
- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2308.11681/) (2023)
- [VideoLLM: Modeling Video Sequence with Large Language Models](https://mlsanity.com/paper/2305.13292/) (2023)
- [Harnessing Large Language Models for Training-free Video Anomaly Detection](https://mlsanity.com/paper/2404.01014/) (2024)
- [Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection](https://mlsanity.com/paper/2108.09896/) (2021)
- [Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models](https://mlsanity.com/paper/2302.05087/) (2023)
- [CLIP-AD: A Language-Guided Staged Dual-Path Model for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2311.00453/) (2023)
- [Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2407.10299/) (2024)
- [FreezeOut: Accelerate Training by Progressively Freezing Layers](https://mlsanity.com/paper/1706.04983/) (2017)
- [SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection](https://mlsanity.com/paper/2207.08003/) (2022)
- [Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection](https://mlsanity.com/paper/2404.09654/) (2024)
- [VideoQA in the Era of LLMs: An Empirical Study](https://mlsanity.com/paper/2408.04223/) (2024)
- [Consistency-based Self-supervised Learning for Temporal Anomaly Localization](https://mlsanity.com/paper/2208.05251/) (2022)
- [ComplexVAD: Detecting Interaction Anomalies in Video](https://mlsanity.com/paper/2501.09733/) (2025)
- [A Critical Study on the Recent Deep Learning Based Semi-Supervised Video Anomaly Detection Methods](https://mlsanity.com/paper/2111.01604/) (2021)
- [AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection](https://mlsanity.com/paper/2504.04495/) (2025)

## Abstract (arXiv)

Video anomaly detection (VAD) is crucial for intelligent surveillance, but a significant challenge lies in identifying complex anomalies, which are events defined by intricate relationships and temporal dependencies among multiple entities rather than by isolated actions. While self-supervised learning (SSL) methods effectively model low-level spatiotemporal patterns, they often struggle to grasp the semantic meaning of these interactions. Conversely, large language models (LLMs) offer powerful contextual reasoning but are computationally expensive for frame-by-frame analysis and lack fine-grained spatial localization. We introduce HyCoVAD, Hybrid Complex Video Anomaly Detection, a hybrid SSL-LLM model that combines a multi-task SSL temporal analyzer with LLM validator. The SSL module is built upon an nnFormer backbone which is a transformer-based model for image segmentation. It is trained with multiple proxy tasks, learns from video frames to identify those suspected of anomaly. The selected frames are then forwarded to the LLM, which enriches the analysis with semantic context by applying structured, rule-based reasoning to validate the presence of anomalies. Experiments on the challenging ComplexVAD dataset show that HyCoVAD achieves a 72.5% frame-level AUC, outperforming existing baselines by 12.5% while reducing LLM computation. We release our interaction anomaly taxonomy, adaptive thresholding protocol, and code to facilitate future research in complex VAD scenarios.