# Transformer-Driven Multimodal Fusion for Explainable Suspiciousness Estimation in Visual Surveillance

arXiv: https://arxiv.org/abs/2512.09311 · HTML: https://mlsanity.com/paper/2512.09311/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [BlazePose: On-device Real-time Body Pose tracking](https://mlsanity.com/paper/2006.10204/) (2020)
- [PyramidBox: A Context-assisted Single Shot Face Detector](https://mlsanity.com/paper/1803.07737/) (2018)
- [Going Deeper in Facial Expression Recognition using Deep Neural Networks](https://mlsanity.com/paper/1511.04110/) (2015)
- [3DV: 3D Dynamic Voxel for Action Recognition in Depth Video](https://mlsanity.com/paper/2005.05501/) (2020)
- [SSH: Single Stage Headless Face Detector](https://mlsanity.com/paper/1708.03979/) (2017)
- [Deep Learning using Linear Support Vector Machines](https://mlsanity.com/paper/1306.0239/) (2013)
- [EXTD: Extremely Tiny Face Detector via Iterative Filter Reuse](https://mlsanity.com/paper/1906.06579/) (2019)
- [Audio-Based Activities of Daily Living (ADL) Recognition with Large-Scale Acoustic Embeddings from Online Videos](https://mlsanity.com/paper/1810.08691/) (2018)
- [Light-Weight RetinaNet for Object Detection](https://mlsanity.com/paper/1905.10011/) (2019)
- [Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing](https://mlsanity.com/paper/1903.02110/) (2019)
- [Face-MagNet: Magnifying Feature Maps to Detect Small Faces](https://mlsanity.com/paper/1803.05258/) (2018)

## Abstract (arXiv)

Suspiciousness estimation is critical for proactive threat detection and ensuring public safety in complex environments. This work introduces a large-scale annotated dataset, USE50k, along with a computationally efficient vision-based framework for real-time suspiciousness analysis. The USE50k dataset contains 65,500 images captured from diverse and uncontrolled environments, such as airports, railway stations, restaurants, parks, and other public areas, covering a broad spectrum of cues including weapons, fire, crowd density, abnormal facial expressions, and unusual body postures. Building on this dataset, we present DeepUSEvision, a lightweight and modular system integrating three key components, i.e., a Suspicious Object Detector based on an enhanced YOLOv12 architecture, dual Deep Convolutional Neural Networks (DCNN-I and DCNN-II) for facial expression and body-language recognition using image and landmark features, and a transformer-based Discriminator Network that adaptively fuses multimodal outputs to yield an interpretable suspiciousness score. Extensive experiments confirm the superior accuracy, robustness, and interpretability of the proposed framework compared to state-of-the-art approaches. Collectively, the USE50k dataset and the DeepUSEvision framework establish a strong and scalable foundation for intelligent surveillance and real-time risk assessment in safety-critical applications.