# Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention

arXiv: https://arxiv.org/abs/2008.06581 · HTML: https://mlsanity.com/paper/2008.06581/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2020.

## Built on by (most cited first, 30 of 37)

- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering](https://mlsanity.com/paper/2310.06238/) (2023)
- [AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization](https://mlsanity.com/paper/2210.05060/) (2022)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Continual Attentive Fusion for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2202.00432/) (2022)
- [Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.13659/) (2024)
- [Joint Multimodal Transformer for Emotion Recognition in the Wild](https://mlsanity.com/paper/2403.10488/) (2024)
- [Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2307.02041/) (2023)
- [CAD -- Contextual Multi-modal Alignment for Dynamic AVQA](https://mlsanity.com/paper/2310.16754/) (2023)
- [NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative](https://mlsanity.com/paper/2406.06499/) (2024)
- [Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking](https://mlsanity.com/paper/2112.07423/) (2021)
- [Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition](https://mlsanity.com/paper/2304.07958/) (2023)
- [Saying the Unseen: Video Descriptions via Dialog Agents](https://mlsanity.com/paper/2106.14069/) (2021)
- [Audio-Visual Speaker Verification via Joint Cross-Attention](https://mlsanity.com/paper/2309.16569/) (2023)
- [UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization](https://mlsanity.com/paper/2404.03179/) (2024)
- [Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention](https://mlsanity.com/paper/2403.04654/) (2024)
- [Variational Structured Attention Networks for Deep Visual Representation Learning](https://mlsanity.com/paper/2103.03510/) (2021)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [Hierarchical Cross-Attention Network for Virtual Try-On](https://mlsanity.com/paper/2411.15542/) (2024)
- [Interactive Multimodal Fusion with Temporal Modeling](https://mlsanity.com/paper/2503.10523/) (2025)
- [ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localization](https://mlsanity.com/paper/2507.09945/) (2025)

## Abstract (arXiv)

The major challenge in audio-visual event localization task lies in how to fuse information from multiple modalities effectively. Recent works have shown that attention mechanism is beneficial to the fusion process. In this paper, we propose a novel joint attention mechanism with multimodal fusion methods for audio-visual event localization. Particularly, we present a concise yet valid architecture that effectively learns representations from multiple modalities in a joint manner. Initially, visual features are combined with auditory features and then turned into joint representations. Next, we make use of the joint representations to attend to visual features and auditory features, respectively. With the help of this joint co-attention, new visual and auditory features are produced, and thus both features can enjoy the mutually improved benefits from each other. It is worth noting that the joint co-attention unit is recursive meaning that it can be performed multiple times for obtaining better joint representations progressively. Extensive experiments on the public AVE dataset have shown that the proposed method achieves significantly better results than the state-of-the-art methods.