# SAVCHOI: Detecting Suspicious Activities using Dense Video Captioning with Human Object Interactions

arXiv: https://arxiv.org/abs/2207.11838 · HTML: https://mlsanity.com/paper/2207.11838/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 24 Jul 2022.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Adversarially Learned One-Class Classifier for Novelty Detection](https://mlsanity.com/paper/1802.09088/) (2018)
- [Graph Convolutional Label Noise Cleaner: Train a Plug-and-play Action Classifier for Anomaly Detection](https://mlsanity.com/paper/1903.07256/) (2019)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021)
- [An Image is Worth 16x16 Words, What is a Video Worth?](https://mlsanity.com/paper/2103.13915/) (2021)
- [TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks](https://mlsanity.com/paper/2011.11479/) (2020)
- [A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer](https://mlsanity.com/paper/2005.08271/) (2020)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Streamlined Dense Video Captioning](https://mlsanity.com/paper/1904.03870/) (2019)
- [Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning](https://mlsanity.com/paper/1909.09944/) (2019)
- [Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video](https://mlsanity.com/paper/2108.03825/) (2021)
- [QAHOI: Query-Based Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2112.08647/) (2021)
- [Anomaly Detection in Video Sequences: A Benchmark and Computational Model](https://mlsanity.com/paper/2106.08570/) (2021)
- [Weakly-supervised Joint Anomaly Detection and Classification](https://mlsanity.com/paper/2108.08996/) (2021)
- [Tell me what you see: A zero-shot action recognition method based on natural language descriptions](https://mlsanity.com/paper/2112.09976/) (2021)

## Built on by (most cited first, 3 of 3)

- [Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges](https://mlsanity.com/paper/2309.13925/) (2023)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [Identity-Aware Human-Object Interaction Motion Captioning](https://mlsanity.com/paper/2608.20690/) (2026)

## Abstract (arXiv)

Detecting suspicious activities in surveillance videos is a longstanding problem in real-time surveillance that leads to difficulties in detecting crimes. Hence, we propose a novel approach for detecting and summarizing suspicious activities in surveillance videos. We have also created ground truth summaries for the UCF-Crime video dataset. We modify a pre-existing approach for this task by leveraging the Human-Object Interaction (HOI) model for the Visual features in the Bi-Modal Transformer. Further, we validate our approach against the existing state-of-the-art algorithms for the Dense Video Captioning task for the ActivityNet Captions dataset. We observe that this formulation for Dense Captioning performs significantly better than other discussed BMT-based approaches for BLEU@1, BLEU@2, BLEU@3, BLEU@4, and METEOR. We further perform a comparative analysis of the dataset and the model to report the findings based on different NMS thresholds (searched using Genetic Algorithms). Here, our formulation outperforms all the models for BLEU@1, BLEU@2, BLEU@3, and most models for BLEU@4 and METEOR falling short of only ADV-INF Global by 25% and 0.5%, respectively.