# Audio-Visual Grouping Network for Sound Localization from Mixtures

arXiv: https://arxiv.org/abs/2303.17056 · HTML: https://mlsanity.com/paper/2303.17056/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Self-Supervised Generation of Spatial Audio for 360 Video](https://mlsanity.com/paper/1809.02587/) (2018)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1608.07017/) (2016)
- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [Dual-modality seq2seq network for audio-visual event localization](https://mlsanity.com/paper/1902.07473/) (2019)
- [2.5D Visual Sound](https://mlsanity.com/paper/1812.04204/) (2018)

## Built on by (most cited first, 30 of 52)

- [AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation](https://mlsanity.com/paper/2305.01836/) (2023)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Text-to-Audio Generation Synchronized with Videos](https://mlsanity.com/paper/2403.07938/) (2024)
- [Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time](https://mlsanity.com/paper/2407.01851/) (2024)
- [Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling](https://mlsanity.com/paper/2312.01017/) (2023)
- [Class-Incremental Grouping Network for Continual Audio-Visual Learning](https://mlsanity.com/paper/2309.05281/) (2023)
- [A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition](https://mlsanity.com/paper/2305.19458/) (2023)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment](https://mlsanity.com/paper/2305.12903/) (2023)
- [Weakly-Supervised Audio-Visual Segmentation](https://mlsanity.com/paper/2311.15080/) (2023)
- [T-VSL: Text-Guided Visual Sound Source Localization in Mixtures](https://mlsanity.com/paper/2404.01751/) (2024)
- [AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation](https://mlsanity.com/paper/2501.07810/) (2025)
- [Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge](https://mlsanity.com/paper/2403.17420/) (2024)
- [MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers](https://mlsanity.com/paper/2406.04930/) (2024)
- [CPM: Class-conditional Prompting Machine for Audio-visual Segmentation](https://mlsanity.com/paper/2407.05358/) (2024)
- [Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization](https://mlsanity.com/paper/2403.03145/) (2024)
- [Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment](https://mlsanity.com/paper/2503.12847/) (2025)
- [Audio-visual Generalized Zero-shot Learning the Easy Way](https://mlsanity.com/paper/2407.13095/) (2024)
- [Continual Audio-Visual Sound Separation](https://mlsanity.com/paper/2411.02860/) (2024)
- [DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap](https://mlsanity.com/paper/2503.12131/) (2025)
- [Decoupled Audio-Visual Dataset Distillation](https://mlsanity.com/paper/2511.17890/) (2025)
- [Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition](https://mlsanity.com/paper/2406.02554/) (2024)
- [Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation](https://mlsanity.com/paper/2406.06163/) (2024)
- [Spherical World-Locking for Audio-Visual Localization in Egocentric Videos](https://mlsanity.com/paper/2408.05364/) (2024)
- [Object-aware Sound Source Localization via Audio-Visual Scene Understanding](https://mlsanity.com/paper/2506.18557/) (2025)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [Complementary and Contrastive Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2510.10051/) (2025)
- [A Critical Assessment of Visual Sound Source Localization Models Including Negative Audio](https://mlsanity.com/paper/2410.01020/) (2024)
- [Aligning Audio-Visual Joint Representations with an Agentic Workflow](https://mlsanity.com/paper/2410.23230/) (2024)
- [Improving Sound Source Localization with Joint Slot Attention on Image and Audio](https://mlsanity.com/paper/2504.15118/) (2025)

## Abstract (arXiv)

Sound source localization is a typical and challenging task that predicts the location of sound sources in a video. Previous single-source methods mainly used the audio-visual association as clues to localize sounding objects in each image. Due to the mixed property of multiple sound sources in the original space, there exist rare multi-source approaches to localizing multiple sources simultaneously, except for one recent work using a contrastive random walk in the graph with images and separated sound as nodes. Despite their promising performance, they can only handle a fixed number of sources, and they cannot learn compact class-aware representations for individual sources. To alleviate this shortcoming, in this paper, we propose a novel audio-visual grouping network, namely AVGN, that can directly learn category-wise semantic features for each source from the input audio mixture and image to localize multiple sources simultaneously. Specifically, our AVGN leverages learnable audio-visual class tokens to aggregate class-aware source features. Then, the aggregated semantic features for each source can be used as guidance to localize the corresponding visual regions. Compared to existing multi-source methods, our new framework can localize a flexible number of sources and disentangle category-aware audio-visual representations for individual sound sources. We conduct extensive experiments on MUSIC, VGGSound-Instruments, and VGG-Sound Sources benchmarks. The results demonstrate that the proposed AVGN can achieve state-of-the-art sounding object localization performance on both single-source and multi-source scenarios. Code is available at urlhttps://github.com/stoneMo/AVGN.