mlsanityData to 7 Oct 2026

Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification

arXiv 2401.04023 ↗First posted 8 Jan 2024cs.CV cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, researchers combine both audio and video signals to deal with challenges where actions are not well represented or captured by visual cues. However, how to effectively leverage the two modalities is still under development. In this work, we develop a multiscale multimodal Transformer (MMT) that leverages hierarchical representation learning. Particularly, MMT is composed of a novel multiscale audio Transformer (MAT) and a multiscale video Transformer [43]. To learn a discriminative cross-modality fusion, we further design multimodal supervised contrastive objectives called audio-video contrastive loss (AVC) and intra-modal contrastive loss (IMC) that robustly align the two modalities. MMT surpasses previous state-of-the-art approaches by 7.3% and 2.1% on Kinetics-Sounds and VGGSound in terms of the top-1 accuracy without external training data. Moreover, the proposed MAT significantly outperforms AST [28] by 22.2%, 4.4% and 4.7% on three public benchmark datasets, and is about 3% more efficient based on the number of FLOPs and 9.8% more efficient based on GPU memory usage.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 65 of its 79 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

65 of 79 references.

References · 65

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
38 more
  1. 2021Multiscale Vision Transformerscited by 896
  2. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  3. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  4. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  6. 2020Rescaling Egocentric Visioncited by 529
  7. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  8. 2017Look, Listen and Learncited by 452
  9. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  10. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  11. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  12. 2017Temporal Relational Reasoning in Videoscited by 324
  13. 2018Deep Audio-Visual Speech Recognitioncited by 318
  14. 2022Masked Autoencoders that Listencited by 271
  15. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  16. 2018The Sound of Pixelscited by 258
  17. 2017Objects that Soundcited by 254
  18. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  19. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  20. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  21. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  22. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  23. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  24. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  25. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  26. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  27. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  28. 2021Localizing Visual Sounds the Hard Waycited by 169
  29. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  30. 2019The Sound of Motionscited by 145
  31. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  32. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  33. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  34. 2019Co-Separating Sounds of Visual Objectscited by 110
  35. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 99
  36. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  37. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  38. 2016Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networkscited by 63

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Comparative Analysis of Image, Video, and Audio Classifiers for Automated News Video Segmentationcited by 1
  2. 2025CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognitioncited by 0
  3. 2025Video-Based MPAA Rating Prediction: An Attention-Driven Hybrid Architecture Using Contrastive Learningcited by 0
  4. 2025OmniFuser: Adaptive Multimodal Fusion for Service-Oriented Predictive Maintenancecited by 0