# A cappella: Audio-visual Singing Voice Separation

arXiv: https://arxiv.org/abs/2104.09946 · HTML: https://mlsanity.com/paper/2104.09946/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 30 of 34)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [Speech2Face: Learning the Face Behind a Voice](https://mlsanity.com/paper/1905.09773/) (2019)
- [Recursive Visual Sound Separation Using Minus-Plus Net](https://mlsanity.com/paper/1908.11602/) (2019)
- [Synthesizing Normalized Faces from Facial Identity Features](https://mlsanity.com/paper/1701.04851/) (2017)
- [Time Domain Audio Visual Speech Separation](https://mlsanity.com/paper/1904.03760/) (2019)
- [Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separations](https://mlsanity.com/paper/1907.01277/) (2019)
- [Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation](https://mlsanity.com/paper/2104.02775/) (2021)
- [On Learning Associations of Faces and Voices](https://mlsanity.com/paper/1805.05553/) (2018)
- [FaceFilter: Audio-visual speech separation using still images](https://mlsanity.com/paper/2005.07074/) (2020)
- [MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation](https://mlsanity.com/paper/1805.02410/) (2018)
- [Meta-learning Extractors for Music Source Separation](https://mlsanity.com/paper/2002.07016/) (2020)
- [Conditioned Source Separation for Music Instrument Performances](https://mlsanity.com/paper/2004.03873/) (2020)
- [Visually Guided Sound Source Separation using Cascaded Opponent Filter Network](https://mlsanity.com/paper/2006.03028/) (2020)
- [Towards Estimating the Upper Bound of Visual-Speech Recognition: The Visual Lip-Reading Feasibility Database](https://mlsanity.com/paper/1704.08028/) (2017)
- [Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments](https://mlsanity.com/paper/1811.02480/) (2018)
- [Deep Learning Based Source Separation Applied To Choir Ensembles](https://mlsanity.com/paper/2008.07645/) (2020)

## Built on by (most cited first, 12 of 12)

- [Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance](https://mlsanity.com/paper/2401.15687/) (2024)
- [VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices](https://mlsanity.com/paper/2204.02090/) (2022)
- [Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization](https://mlsanity.com/paper/2505.11217/) (2025)
- [VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer](https://mlsanity.com/paper/2203.04099/) (2022)
- [An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits](https://mlsanity.com/paper/2212.10744/) (2022)
- [Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion](https://mlsanity.com/paper/2502.07203/) (2025)
- [PianoVAM: A Multimodal Piano Performance Dataset](https://mlsanity.com/paper/2509.08800/) (2025)
- [Towards Robust Real-time Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2112.09060/) (2021)
- [SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge Evaluation Plan](https://mlsanity.com/paper/2405.05244/) (2024)
- [TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion](https://mlsanity.com/paper/2401.14185/) (2024)
- [VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categories](https://mlsanity.com/paper/2604.10542/) (2026)
- [MambaVoice: Lightweight Audiovisual Singing Voice Separation Via A Hybrid Mamba-Transformer Model](https://mlsanity.com/paper/2609.26635/) (2026)

## Abstract (arXiv)

The task of isolating a target singing voice in music videos has useful applications. In this work, we explore the single-channel singing voice separation problem from a multimodal perspective, by jointly learning from audio and visual modalities. To do so, we present Acappella, a dataset spanning around 46 hours of a cappella solo singing videos sourced from YouTube. We also propose an audio-visual convolutional network based on graphs which achieves state-of-the-art singing voice separation results on our dataset and compare it against its audio-only counterpart, U-Net, and a state-of-the-art audio-visual speech separation model. We evaluate the models in the following challenging setups: i) presence of overlapping voices in the audio mixtures, ii) the target voice set to lower volume levels in the mix, and iii) combination of i) and ii). The third one being the most challenging evaluation setup. We demonstrate that our model outperforms the baseline models in the singing voice separation task in the most challenging evaluation setup. The code, the pre-trained models, and the dataset are publicly available at https://ipcv.github.io/Acappella/able at https://ipcv.github.io/Acappella/