# Weakly-Supervised Audio-Visual Segmentation

arXiv: https://arxiv.org/abs/2311.15080 · HTML: https://mlsanity.com/paper/2311.15080/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 75 references that are in the map.
- First posted on arXiv 25 Nov 2023.

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [A Simple Pooling-Based Design for Real-Time Salient Object Detection](https://mlsanity.com/paper/1904.09569/) (2019)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation](https://mlsanity.com/paper/1603.06098/) (2016)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation](https://mlsanity.com/paper/1502.02734/) (2015)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference](https://mlsanity.com/paper/1902.10421/) (2019)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi- Supervised Semantic Segmentation](https://mlsanity.com/paper/1805.04574/) (2018)
- [Weakly-Supervised Semantic Segmentation via Sub-category Exploration](https://mlsanity.com/paper/2008.01183/) (2020)

## Built on by (most cited first, 19 of 19)

- [Text-to-Audio Generation Synchronized with Videos](https://mlsanity.com/paper/2403.07938/) (2024)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling](https://mlsanity.com/paper/2312.01017/) (2023)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap](https://mlsanity.com/paper/2503.12131/) (2025)
- [Semantic Grouping Network for Audio Source Separation](https://mlsanity.com/paper/2407.03736/) (2024)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [Complementary and Contrastive Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2510.10051/) (2025)
- [Aligning Audio-Visual Joint Representations with an Agentic Workflow](https://mlsanity.com/paper/2410.23230/) (2024)
- [Multi-scale Multi-instance Visual Sound Localization and Segmentation](https://mlsanity.com/paper/2409.00486/) (2024)
- [Measuring Sound Symbolism in Audio-visual Models](https://mlsanity.com/paper/2409.12306/) (2024)
- [From Waveforms to Pixels: A Survey on Audio-Visual Segmentation](https://mlsanity.com/paper/2508.03724/) (2025)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [Unified Video-Language Pre-training with Synchronized Audio](https://mlsanity.com/paper/2405.07202/) (2024)
- [3D Audio-Visual Segmentation](https://mlsanity.com/paper/2411.02236/) (2024)
- [AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting](https://mlsanity.com/paper/2506.01015/) (2025)
- [OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Models](https://mlsanity.com/paper/2505.01448/) (2025)
- [Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows](https://mlsanity.com/paper/2603.08126/) (2026)
- [Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation](https://mlsanity.com/paper/2603.08967/) (2026)

## Abstract (arXiv)

Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision. However, these pixel-level masks are expensive and not available in all cases. In this work, we aim to simplify the supervision as the instance-level annotation, i.e., weakly-supervised audio-visual segmentation. We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.