mlsanityData to 7 Oct 2026

MAiVAR: Multimodal Audio-Image and Video Action Recognizer

arXiv 2209.04780 ↗First posted 11 Sep 2022cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNNs can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 11 Sep 2022.

Builds on

24 of 35 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  10. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  11. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  12. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
12 more
  1. 2017Temporal Relational Reasoning in Videoscited by 324
  2. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  3. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  4. 2017Compressed Video Action Recognitioncited by 102
  5. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 89
  6. 2019Gate-Shift Networks for Video Action Recognitioncited by 76
  7. 2017ActionVLAD: Learning spatio-temporal aggregation for action classificationcited by 73
  8. 2021Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognitioncited by 31
  9. 2018StNet: Local and Global Spatial-Temporal Modeling for Action Recognitioncited by 28
  10. 2019Collaborative Spatio-temporal Feature Learning for Video Action Recognitioncited by 24
  11. 2021NTIRE 2021 Challenge on Perceptual Image Quality Assessmentcited by 18
  12. 2017AENet: Learning Deep Audio Features for Video Analysiscited by 13

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024ASTRA: An Action Spotting TRAnsformer for Soccer Videoscited by 16
  2. 2024From CNNs to Transformers in Multimodal Human Action Recognition: A Surveycited by 7
  3. 2023MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformerscited by 4