# Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification

arXiv: https://arxiv.org/abs/2408.14441 · HTML: https://mlsanity.com/paper/2408.14441/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 71 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [Attention-Based Multimodal Fusion for Video Description](https://mlsanity.com/paper/1701.03126/) (2017)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning](https://mlsanity.com/paper/2008.05789/) (2020)
- [Learnable pooling with Context Gating for video classification](https://mlsanity.com/paper/1706.06905/) (2017)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing](https://mlsanity.com/paper/2111.12374/) (2021)
- [Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition](https://mlsanity.com/paper/2110.10101/) (2021)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [Self-Supervised Learning of Video-Induced Visual Invariances](https://mlsanity.com/paper/1912.02783/) (2019)
- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [Temporal and cross-modal attention for audio-visual zero-shot learning](https://mlsanity.com/paper/2207.09966/) (2022)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)

## Built on by (most cited first, 3 of 3)

- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [Efficient Video-to-Audio Generation via Multiple Foundation Models Mapper](https://mlsanity.com/paper/2509.04957/) (2025)
- [Multimodal Confidence Modeling in Audio-Visual Quality Assessment](https://mlsanity.com/paper/2605.01219/) (2026)

## Abstract (arXiv)

Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a compact model architecture specifically designed to capture intricate audio-visual relationships in video data. Through extensive experiments on the challenging YouTube-8M dataset, we demonstrate that Attend-Fusion achieves an F1 score of 75.64% with only 72M parameters, which is comparable to the performance of larger baseline models such as Fully-Connected Late Fusion (75.96% F1 score, 341M parameters). Attend-Fusion achieves similar performance to the larger baseline model while reducing the model size by nearly 80%, highlighting its efficiency in terms of model complexity. Our work demonstrates that the Attend-Fusion model effectively combines audio and visual information for video classification, achieving competitive performance with significantly reduced model size. This approach opens new possibilities for deploying high-performance video understanding systems in resource-constrained environments across various applications.