# MissionGNN: Hierarchical Multimodal GNN-based Weakly Supervised Video Anomaly Recognition with Mission-Specific Knowledge Graph Generation

arXiv: https://arxiv.org/abs/2406.18815 · HTML: https://mlsanity.com/paper/2406.18815/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 61 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning](https://mlsanity.com/paper/2101.10030/) (2021)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction](https://mlsanity.com/paper/2108.06852/) (2021)

## Built on by (most cited first, 14 of 14)

- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Reality](https://mlsanity.com/paper/2508.09185/) (2025)
- [PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation](https://mlsanity.com/paper/2410.22623/) (2024)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [Evaluation of Vision-LLMs in Surveillance Video](https://mlsanity.com/paper/2510.23190/) (2025)
- [Recoverable Anonymization for Pose Estimation: A Privacy-Enhancing Approach](https://mlsanity.com/paper/2409.02715/) (2024)
- [Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning](https://mlsanity.com/paper/2411.09072/) (2024)
- [Is Video Anomaly Detection Misframed? Evidence from LLM-Based and Multi-Scene Models](https://mlsanity.com/paper/2605.12725/) (2026)
- [Detection, Retrieval, and Explanation Unified: A Violence Detection System Based on Knowledge Graphs and GAT](https://mlsanity.com/paper/2501.06224/) (2025)
- [PacketCLIP: Multi-Modal Embedding of Network Traffic and Language for Cybersecurity Reasoning](https://mlsanity.com/paper/2503.03747/) (2025)
- [MissionHD: Hyperdimensional Refinement of Distribution-Deficient Reasoning Graphs for Video Anomaly Detection](https://mlsanity.com/paper/2508.14746/) (2025)
- [TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI](https://mlsanity.com/paper/2603.22867/) (2026)
- [FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence](https://mlsanity.com/paper/2605.22868/) (2026)
- [Localizing to Debias: A Patch-Level Benchmark and Baseline for Weakly Supervised Spatial Anomaly Detection](https://mlsanity.com/paper/2608.12045/) (2026)

## Abstract (arXiv)

In the context of escalating safety concerns across various domains, the tasks of Video Anomaly Detection (VAD) and Video Anomaly Recognition (VAR) have emerged as critically important for applications in intelligent surveillance, evidence investigation, violence alerting, etc. These tasks, aimed at identifying and classifying deviations from normal behavior in video data, face significant challenges due to the rarity of anomalies which leads to extremely imbalanced data and the impracticality of extensive frame-level data annotation for supervised learning. This paper introduces a novel hierarchical graph neural network (GNN) based model MissionGNN that addresses these challenges by leveraging a state-of-the-art large language model and a comprehensive knowledge graph for efficient weakly supervised learning in VAR. Our approach circumvents the limitations of previous methods by avoiding heavy gradient computations on large multimodal models and enabling fully frame-level training without fixed video segmentation. Utilizing automated, mission-specific knowledge graph generation, our model provides a practical and efficient solution for real-time video analysis without the constraints of previous segmentation-based or multimodal approaches. Experimental validation on benchmark datasets demonstrates our model's performance in VAD and VAR, highlighting its potential to redefine the landscape of anomaly detection and recognition in video surveillance systems. The code is available here: https://github.com/c0510gy/MissionGNN.