mlsanityData to 7 Oct 2026

A Toolchain for Comprehensive Audio/Video Analysis Using Deep Learning Based Multimodal Approach (A use case of riot or violent context detection)

arXiv 2407.03110 ↗First posted 2 May 2024cs.SD cs.AI eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a toolchain for a comprehensive audio/video analysis by leveraging deep learning based multimodal approach. To this end, different specific tasks of Speech to Text (S2T), Acoustic Scene Classification (ASC), Acoustic Event Detection (AED), Visual Object Detection (VOD), Image Captioning (IC), and Video Captioning (VC) are conducted and integrated into the toolchain. By combining individual tasks and analyzing both audio & visual data extracted from input video, the toolchain offers various audio/video-based applications: Two general applications of audio/video clustering, comprehensive audio/video summary and a specific application of riot or violent context detection. Furthermore, the toolchain presents a flexible and adaptable architecture that is effective to integrate new models for further audio/video-based applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 18 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

13 of 18 references.

References · 13

  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  3. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  4. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  5. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  6. 2019ViSiL: Fine-grained Spatio-Temporal Video Similarity Learningcited by 41
  7. 2018FIVR: Fine-grained Incident Video Retrievalcited by 26
  8. 2020A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysiscited by 16
  9. 2022Wider or Deeper Neural Network Architecture for Acoustic Scene Classification with Mismatched Recording Devicescited by 7
  10. 2021An Audio-Visual Dataset and Deep Learning Frameworks for Crowded Scene Classificationcited by 5
  11. 2021An Audio-Based Deep Learning Framework For BBC Television Programme Classificationcited by 2
  12. 2023Deep Learning Based Multimodal with Two-phase Training Strategy for Daily Life Video Classificationcited by 1
1 more
  1. 2023MSEVA : A System for Multimodal Short Videos Emotion Visual Analysiscited by 1

Built on by

Not cited yet.