mlsanityData to 7 Oct 2026

Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation

arXiv 2310.11713 ↗First posted 18 Oct 2023cs.CV cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds lacking visible cues. This paper introduces a novel "Audio-Visual Scene-Aware Separation" (AVSA-Sep) framework. It includes a semantic parser for visible and invisible sounds and a separator for scene-informed separation. AVSA-Sep successfully separates both sound types, with joint training and cross-modal alignment enhancing effectiveness.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

37 of 56 references.

References · 37

  1. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2018The Sound of Pixelscited by 258
  4. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  5. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  6. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  7. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  8. 2022Audio-Visual Segmentationcited by 148
  9. 2019The Sound of Motionscited by 145
  10. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  11. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  12. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
25 more
  1. 2020Music Gesture for Visual Sound Separationcited by 112
  2. 2019Co-Separating Sounds of Visual Objectscited by 110
  3. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 97
  4. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  5. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  6. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  7. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  8. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  9. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  10. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  11. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  12. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  13. 2022Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsingcited by 29
  14. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  15. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  16. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  17. 2022AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separationcited by 24
  18. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  19. 2023Language-Guided Audio-Visual Source Separation via Trimodal Consistencycited by 20
  20. 2022Temporal and cross-modal attention for audio-visual zero-shot learningcited by 19
  21. 2022Active Audio-Visual Separation of Dynamic Sound Sourcescited by 16
  22. 2020Visually Guided Sound Source Separation using Cascaded Opponent Filter Networkcited by 15
  23. 2022VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformercited by 14
  24. 2022Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separationcited by 11
  25. 2022Sound and Visual Representation Learning with Multiple Pretraining Taskscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Continual Audio-Visual Sound Separationcited by 7
  2. 2024Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognitioncited by 5
  3. 2025Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Modelcited by 2
  4. 2025MARS-Sep: Multimodal-Aligned Reinforced Sound Separationcited by 1