mlsanityData to 7 Oct 2026

Audio-Visual Segmentation

arXiv 2207.05042 ↗First posted 11 Jul 2022cs.CV cs.MM cs.SDcited by 148

Not on the map. Why?

Abstract (arXiv)

We propose to explore a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise annotations for the sounding objects in audible videos. Two settings are studied with this benchmark: 1) semi-supervised audio-visual segmentation with a single sound source and 2) fully-supervised audio-visual segmentation with multiple sound sources. To deal with the AVS problem, we propose a novel method that uses a temporal pixel-wise audio-visual interaction module to inject audio semantics as guidance for the visual segmentation process. We also design a regularization loss to encourage the audio-visual mapping during training. Quantitative and qualitative experiments on the AVSBench compare our approach to several existing methods from related tasks, demonstrating that the proposed method is promising for building a bridge between the audio and pixel-wise visual semantics. Code is available at https://github.com/OpenNLPLab/AVSBench.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

39 of 53 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2017Mask R-CNNcited by 7,259
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  10. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
27 more
  1. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  2. 2019Panoptic Feature Pyramid Networkscited by 644
  3. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  4. 2017Look, Listen and Learncited by 452
  5. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  6. 2018The Sound of Pixelscited by 258
  7. 2017Objects that Soundcited by 254
  8. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  9. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  10. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  11. 2021Localizing Visual Sounds the Hard Waycited by 169
  12. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  13. 2019The Sound of Motionscited by 145
  14. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  15. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  16. 2019Co-Separating Sounds of Visual Objectscited by 110
  17. 2021SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentationcited by 106
  18. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  19. 2019Dual-modality seq2seq network for audio-visual event localizationcited by 72
  20. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  21. 2020Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learningcited by 63
  22. 2021Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Predictioncited by 57
  23. 2020Making a Case for 3D Convolutions for Object Segmentation in Videoscited by 56
  24. 2019Self-Supervised Audio-Visual Co-Segmentationcited by 44
  25. 2021MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsingcited by 39
  26. 2020Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attentioncited by 37
  27. 2021GETAM: Gradient-weighted Element-wise Transformer Attention Map for Weakly-supervised Semantic segmentationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 40; 2024: 29; 2025: 42; 2026: 29.

148 papers.

Sort:

By area:

  1. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  2. 2022The Devil in Linear Transformercited by 98
  3. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  4. 2023Audio-Visual Segmentation with Semanticscited by 73
  5. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  6. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 66
  7. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  8. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  9. 2023AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentationcited by 52
  10. 2023CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentationcited by 52
  11. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  12. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
38 more
  1. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  2. 2023Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learningcited by 46
  3. 2023A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineeringcited by 45
  4. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  5. 2023Annotation-free Audio-Visual Segmentationcited by 44
  6. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  7. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  8. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  9. 2023Improving Audio-Visual Segmentation with Bidirectional Generationcited by 36
  10. 2023Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizercited by 35
  11. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  12. 2023Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentationcited by 33
  13. 2023Audio-aware Query-enhanced Transformer for Audio-Visual Segmentationcited by 32
  14. 2023Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semanticscited by 32
  15. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 32
  16. 2023Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Taskscited by 31
  17. 2023Contrastive Conditional Latent Diffusion for Audio-visual Segmentationcited by 30
  18. 2023BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledgecited by 30
  19. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  20. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  21. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  22. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  23. 2024Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecturecited by 28
  24. 2023Discovering Sounding Objects by Audio Queries for Audio Visual Segmentationcited by 27
  25. 2024Audio-Visual Segmentation via Unlabeled Frame Exploitationcited by 27
  26. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  27. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  28. 2023Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentationcited by 25
  29. 2024Label-anticipated Event Disentanglement for Audio-Visual Video Parsingcited by 25
  30. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  31. 2025Multimodal Referring Segmentation: A Surveycited by 22
  32. 2023Audio-Visual Instance Segmentationcited by 21
  33. 2024Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentationcited by 21
  34. 2024Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labelingcited by 20
  35. 2023QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decompositioncited by 19
  36. 2023Weakly-Supervised Audio-Visual Segmentationcited by 19
  37. 2023Improving Audio-Visual Video Parsing with Pseudo Visual Labelscited by 18
  38. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 18

50 most cited of 148 All 148 (JSON)