# Audiovisual Singing Voice Separation

arXiv: https://arxiv.org/abs/2107.00231 · HTML: https://mlsanity.com/paper/2107.00231/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 1 Jul 2021.

## Builds on (most cited first, 27 of 27)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds](https://mlsanity.com/paper/2011.01143/) (2020)
- [Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed](https://mlsanity.com/paper/1909.01174/) (2019)
- [The 2018 Signal Separation Evaluation Campaign](https://mlsanity.com/paper/1804.06267/) (2018)
- [Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.00944/) (2017)
- [D3Net: Densely connected multidilated DenseNet for music source separation](https://mlsanity.com/paper/2010.01733/) (2020)
- [MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation](https://mlsanity.com/paper/1805.02410/) (2018)
- [End-to-end music source separation: is it possible in the waveform domain?](https://mlsanity.com/paper/1810.12187/) (2018)
- [Multi-scale Multi-band DenseNets for Audio Source Separation](https://mlsanity.com/paper/1706.09588/) (2017)
- [LaSAFT: Latent Source Attentive Frequency Transformation for Conditioned Source Separation](https://mlsanity.com/paper/2010.11631/) (2020)
- [Deep Clustering and Conventional Networks for Music Separation: Stronger Together](https://mlsanity.com/paper/1611.06265/) (2016)
- [Denoising Auto-encoder with Recurrent Skip Connections and Residual Regression for Music Source Separation](https://mlsanity.com/paper/1807.01898/) (2018)
- [WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separation](https://mlsanity.com/paper/1911.09783/) (2019)
- [Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention](https://mlsanity.com/paper/2106.09669/) (2021)
- [CatNet: music source separation system with mix-audio augmentation](https://mlsanity.com/paper/2102.09966/) (2021)
- [Vision-based Detection of Acoustic Timed Events: a Case Study on Clarinet Note Onsets](https://mlsanity.com/paper/1706.09556/) (2017)
- [Investigating U-Nets with various Intermediate Blocks for Spectrogram-based Singing Voice Separation](https://mlsanity.com/paper/1912.02591/) (2019)

## Built on by (most cited first, 3 of 3)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge Evaluation Plan](https://mlsanity.com/paper/2405.05244/) (2024)
- [MambaVoice: Lightweight Audiovisual Singing Voice Separation Via A Hybrid Mamba-Transformer Model](https://mlsanity.com/paper/2609.26635/) (2026)

## Abstract (arXiv)

Separating a song into vocal and accompaniment components is an active research topic, and recent years witnessed an increased performance from supervised training using deep learning techniques. We propose to apply the visual information corresponding to the singers' vocal activities to further improve the quality of the separated vocal signals. The video frontend model takes the input of mouth movement and fuses it into the feature embeddings of an audio-based separation framework. To facilitate the network to learn audiovisual correlation of singing activities, we add extra vocal signals irrelevant to the mouth movement to the audio mixture during training. We create two audiovisual singing performance datasets for training and evaluation, respectively, one curated from audition recordings on the Internet, and the other recorded in house. The proposed method outperforms audio-based methods in terms of separation quality on most test recordings. This advantage is especially pronounced when there are backing vocals in the accompaniment, which poses a great challenge for audio-only methods.