# Audio-Visual Spatial Integration and Recursive Attention for Robust Sound Source Localization

arXiv: https://arxiv.org/abs/2308.06087 · HTML: https://mlsanity.com/paper/2308.06087/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 65 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [Recursive Contour Saliency Blending Network for Accurate Salient Object Detection](https://mlsanity.com/paper/2105.13865/) (2021)
- [Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization](https://mlsanity.com/paper/2211.03019/) (2022)
- [Reverberant Sound Localization with a Robot Head Based on Direct-Path Relative Transfer Function](https://mlsanity.com/paper/2012.03574/) (2020)
- [Multi-stage Object Detection with Group Recursive Learning](https://mlsanity.com/paper/1608.05159/) (2016)
- [Iterative Deep Convolutional Encoder-Decoder Network for Medical Image Segmentation](https://mlsanity.com/paper/1708.03431/) (2017)

## Built on by (most cited first, 5 of 5)

- [Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge](https://mlsanity.com/paper/2403.17420/) (2024)
- [Object-aware Sound Source Localization via Audio-Visual Scene Understanding](https://mlsanity.com/paper/2506.18557/) (2025)
- [Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality](https://mlsanity.com/paper/2407.16171/) (2024)
- [Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup](https://mlsanity.com/paper/2503.02284/) (2025)
- [Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning](https://mlsanity.com/paper/2604.06824/) (2026)

## Abstract (arXiv)

The objective of the sound source localization task is to enable machines to detect the location of sound-making objects within a visual scene. While the audio modality provides spatial cues to locate the sound source, existing approaches only use audio as an auxiliary role to compare spatial regions of the visual modality. Humans, on the other hand, utilize both audio and visual modalities as spatial cues to locate sound sources. In this paper, we propose an audio-visual spatial integration network that integrates spatial cues from both modalities to mimic human behavior when detecting sound-making objects. Additionally, we introduce a recursive attention network to mimic human behavior of iterative focusing on objects, resulting in more accurate attention regions. To effectively encode spatial information from both modalities, we propose audio-visual pair matching loss and spatial region alignment loss. By utilizing the spatial cues of audio-visual modalities and recursively focusing objects, our method can perform more robust sound source localization. Comprehensive experimental results on the Flickr SoundNet and VGG-Sound Source datasets demonstrate the superiority of our proposed method over existing approaches. Our code is available at: https://github.com/VisualAIKHU/SIRA-SSL