mlsanityData to 7 Oct 2026

Weakly-Supervised Audio-Visual Segmentation

arXiv 2311.15080 ↗First posted 25 Nov 2023cs.CV cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision. However, these pixel-level masks are expensive and not available in all cases. In this work, we aim to simplify the supervision as the instance-level annotation, i.e., weakly-supervised audio-visual segmentation. We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 64 of its 75 references that are in the map. First posted on arXiv 25 Nov 2023.

Builds on

64 of 75 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  3. 2019Panoptic Feature Pyramid Networkscited by 644
  4. 2017Look, Listen and Learncited by 452
  5. 2016ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentationcited by 330
  6. 2020Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentationcited by 299
  7. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
  8. 2018The Sound of Pixelscited by 258
  9. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  10. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  11. 2015BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentationcited by 238
  12. 2020Causal Intervention for Weakly-Supervised Semantic Segmentationcited by 229
38 more
  1. 2016Simple Does It: Weakly Supervised Instance and Semantic Segmentationcited by 226
  2. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  3. 2019A Simple Pooling-Based Design for Real-Time Salient Object Detectioncited by 211
  4. 2017Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approachcited by 201
  5. 2016Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentationcited by 198
  6. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  7. 2015Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentationcited by 189
  8. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  9. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  10. 2021Localizing Visual Sounds the Hard Waycited by 169
  11. 2018Tell Me Where to Look: Guided Attention Inference Networkcited by 168
  12. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  13. 2019FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inferencecited by 162
  14. 2022Audio-Visual Segmentationcited by 148
  15. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  16. 2018Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi- Supervised Semantic Segmentationcited by 145
  17. 2019The Sound of Motionscited by 145
  18. 2020Weakly-Supervised Semantic Segmentation via Sub-category Explorationcited by 145
  19. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  20. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  21. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  22. 2020Music Gesture for Visual Sound Separationcited by 112
  23. 2019Co-Separating Sounds of Visual Objectscited by 110
  24. 2018Learning random-walk label propagation for weakly-supervised semantic segmentationcited by 101
  25. 2018Self-Erasing Network for Integral Object Attentioncited by 95
  26. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  27. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  28. 2022Localizing Visual Sounds the Easy Waycited by 81
  29. 2021Robust Audio-Visual Instance Discriminationcited by 80
  30. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  31. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  32. 20182.5D Visual Soundcited by 63
  33. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  34. 2019Self-Supervised Difference Detection for Weakly-Supervised Semantic Segmentationcited by 58
  35. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  36. 2023AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentationcited by 52
  37. 2019Self-Supervised Audio-Visual Co-Segmentationcited by 44
  38. 2021Background-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentationcited by 41

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 8; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  2. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  3. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  4. 2025Multimodal Referring Segmentation: A Surveycited by 22
  5. 2025DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gapcited by 7
  6. 2024Semantic Grouping Network for Audio Source Separationcited by 5
  7. 2024Aligning Audio-Visual Joint Representations with an Agentic Workflowcited by 4
  8. 2025Implicit Counterfactual Learning for Audio-Visual Segmentationcited by 4
  9. 2025Complementary and Contrastive Learning for Audio-Visual Segmentationcited by 4
  10. 2024Multi-scale Multi-instance Visual Sound Localization and Segmentationcited by 3
  11. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  12. 2025From Waveforms to Pixels: A Survey on Audio-Visual Segmentationcited by 3
7 more
  1. 2025Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentationcited by 3
  2. 2024Unified Video-Language Pre-training with Synchronized Audiocited by 2
  3. 20243D Audio-Visual Segmentationcited by 2
  4. 2025AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Promptingcited by 2
  5. 2025OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Modelscited by 1
  6. 2026Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flowscited by 1
  7. 2026Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentationcited by 1