# AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation

arXiv: https://arxiv.org/abs/2207.10141 · HTML: https://mlsanity.com/paper/2207.10141/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 59 references that are in the map.
- First posted on arXiv 20 Jul 2022.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning](https://mlsanity.com/paper/2008.05789/) (2020)
- [Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds](https://mlsanity.com/paper/2011.01143/) (2020)
- [Recursive Visual Sound Separation Using Minus-Plus Net](https://mlsanity.com/paper/1908.11602/) (2019)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Self-Supervised Audio-Visual Co-Segmentation](https://mlsanity.com/paper/1904.09013/) (2019)
- [Universal Sound Separation](https://mlsanity.com/paper/1905.03330/) (2019)
- [Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.00944/) (2017)
- [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018)

## Built on by (most cited first, 24 of 24)

- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)
- [iQuery: Instruments as Queries for Audio-Visual Sound Separation](https://mlsanity.com/paper/2212.03814/) (2022)
- [Audio-Visual Segmentation via Unlabeled Frame Exploitation](https://mlsanity.com/paper/2403.11074/) (2024)
- [Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation](https://mlsanity.com/paper/2303.11329/) (2023)
- [Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective](https://mlsanity.com/paper/2306.00595/) (2023)
- [Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2307.02041/) (2023)
- [High-Quality Visually-Guided Sound Separation from Diverse Categories](https://mlsanity.com/paper/2308.00122/) (2023)
- [Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization](https://mlsanity.com/paper/2403.03145/) (2024)
- [LAVSS: Location-Guided Audio-Visual Spatial Audio Separation](https://mlsanity.com/paper/2310.20446/) (2023)
- [OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup](https://mlsanity.com/paper/2410.21269/) (2024)
- [Continual Audio-Visual Sound Separation](https://mlsanity.com/paper/2411.02860/) (2024)
- [ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior](https://mlsanity.com/paper/2505.05657/) (2025)
- [Optimal Condition Training for Target Source Separation](https://mlsanity.com/paper/2211.05927/) (2022)
- [UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures](https://mlsanity.com/paper/2305.20054/) (2023)
- [Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation](https://mlsanity.com/paper/2310.11713/) (2023)
- [SAVE: Segment Audio-Visual Easy way using Segment Anything Model](https://mlsanity.com/paper/2407.02004/) (2024)
- [SonoWorld: From One Image to a 3D Audio-Visual Scene](https://mlsanity.com/paper/2603.28757/) (2026)
- [High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling](https://mlsanity.com/paper/2509.22063/) (2025)
- [Egocentric Audio-Visual Noise Suppression](https://mlsanity.com/paper/2211.03643/) (2022)
- [MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR](https://mlsanity.com/paper/2603.10465/) (2026)
- [Complete and separate: Conditional separation with missing target source attribute completion](https://mlsanity.com/paper/2307.14609/) (2023)
- [Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separator](https://mlsanity.com/paper/2504.18283/) (2025)
- [PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation](https://mlsanity.com/paper/2609.04867/) (2026)

## Abstract (arXiv)

We introduce AudioScopeV2, a state-of-the-art universal audio-visual on-screen sound separation system which is capable of learning to separate sounds and associate them with on-screen objects by looking at in-the-wild videos. We identify several limitations of previous work on audio-visual on-screen sound separation, including the coarse resolution of spatio-temporal attention, poor convergence of the audio separation model, limited variety in training and evaluation data, and failure to account for the trade off between preservation of on-screen sounds and suppression of off-screen sounds. We provide solutions to all of these issues. Our proposed cross-modal and self-attention network architectures capture audio-visual dependencies at a finer resolution over time, and we also propose efficient separable variants that are capable of scaling to longer videos without sacrificing much performance. We also find that pre-training the separation model only on audio greatly improves results. For training and evaluation, we collected new human annotations of onscreen sounds from a large database of in-the-wild videos (YFCC100M). This new dataset is more diverse and challenging. Finally, we propose a calibration procedure that allows exact tuning of on-screen reconstruction versus off-screen suppression, which greatly simplifies comparing performance between models with different operating points. Overall, our experimental results show marked improvements in on-screen separation performance under much more general conditions than previous methods with minimal additional computational complexity.