mlsanityData to 7 Oct 2026

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

arXiv 2310.20446 ↗First posted 31 Oct 2023cs.SD cs.CV cs.MMcited by 8

Not on the map. Why?

Abstract (arXiv)

Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, not where it is located. This can be a problem in VR/AR scenarios, where listeners need to be able to distinguish between similar audio sources located in different directions. To address this limitation, we have generalized MAVS to spatial audio separation and proposed LAVSS: a location-guided audio-visual spatial audio separator. LAVSS is inspired by the correlation between spatial audio and visual location. We introduce the phase difference carried by binaural audio as spatial cues, and we utilize positional representations of sounding objects as additional modality guidance. We also leverage multi-level cross-modal attention to perform visual-positional collaboration with audio features. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation. This exploits the correlation between monaural and binaural channels. Experiments on the FAIR-Play dataset demonstrate the superiority of the proposed LAVSS over existing benchmarks of audio-visual separation. Our project page: https://yyx666660.github.io/LAVSS/.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 55 of its 81 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

55 of 81 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  6. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  7. 2022AudioGen: Textually Guided Audio Generationcited by 292
  8. 2018The Sound of Pixelscited by 258
  9. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  10. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  11. 2019The Sound of Motionscited by 145
  12. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133
38 more
  1. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  2. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  3. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  4. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  5. 2020Music Gesture for Visual Sound Separationcited by 112
  6. 2019Co-Separating Sounds of Visual Objectscited by 110
  7. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  8. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  9. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  10. 2020Deep Audio-Visual Learning: A Surveycited by 68
  11. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  12. 2022Learning Neural Acoustic Fieldscited by 65
  13. 20182.5D Visual Soundcited by 63
  14. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  15. 2020Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learningcited by 63
  16. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  17. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  18. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  19. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  20. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  21. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  22. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  23. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  24. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  25. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  26. 2017CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generationcited by 31
  27. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  28. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  29. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  30. 2018Scene-Aware Audio for 360textdegree Videoscited by 25
  31. 2022AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separationcited by 24
  32. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  33. 2021Visually Guided Sound Source Separation and Localization using Self-Supervised Motion Representationscited by 18
  34. 2021Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attentioncited by 17
  35. 2022Active Audio-Visual Separation of Dynamic Sound Sourcescited by 16
  36. 2020Visually Guided Sound Source Separation using Cascaded Opponent Filter Networkcited by 15
  37. 2021Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generationcited by 14
  38. 2022VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformercited by 14

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  2. 2024Continual Audio-Visual Sound Separationcited by 7
  3. 2024Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speechcited by 4
  4. 2024SOAF: Scene Occlusion-aware Neural Acoustic Fieldcited by 3
  5. 2025SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Bindingcited by 3
  6. 2025UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsingcited by 2
  7. 2025MARS-Sep: Multimodal-Aligned Reinforced Sound Separationcited by 1
  8. 2026SceneBind: Binding What and Where Across Vision, Audio and Languagecited by 0