mlsanityData to 7 Oct 2026

A cappella: Audio-visual Singing Voice Separation

arXiv 2104.09946 ↗First posted 20 Apr 2021cs.SD cs.LG eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

The task of isolating a target singing voice in music videos has useful applications. In this work, we explore the single-channel singing voice separation problem from a multimodal perspective, by jointly learning from audio and visual modalities. To do so, we present Acappella, a dataset spanning around 46 hours of a cappella solo singing videos sourced from YouTube. We also propose an audio-visual convolutional network based on graphs which achieves state-of-the-art singing voice separation results on our dataset and compare it against its audio-only counterpart, U-Net, and a state-of-the-art audio-visual speech separation model. We evaluate the models in the following challenging setups: i) presence of overlapping voices in the audio mixtures, ii) the target voice set to lower volume levels in the mix, and iii) combination of i) and ii). The third one being the most challenging evaluation setup. We demonstrate that our model outperforms the baseline models in the singing voice separation task in the most challenging evaluation setup. The code, the pre-trained models, and the dataset are publicly available at https://ipcv.github.io/Acappella/able at https://ipcv.github.io/Acappella/

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

34 of 47 references.

References · 34

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  5. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  6. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
  7. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  8. 2018The Sound of Pixelscited by 258
  9. 2019The Sound of Motionscited by 145
  10. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  11. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  12. 2020Music Gesture for Visual Sound Separationcited by 112
22 more
  1. 2019Co-Separating Sounds of Visual Objectscited by 110
  2. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  3. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  4. 2019Speech2Face: Learning the Face Behind a Voicecited by 55
  5. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  6. 2017Synthesizing Normalized Faces from Facial Identity Featurescited by 33
  7. 2019Time Domain Audio Visual Speech Separationcited by 30
  8. 2019Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separationscited by 27
  9. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  10. 2018On Learning Associations of Faces and Voicescited by 26
  11. 2020FaceFilter: Audio-visual speech separation using still imagescited by 25
  12. 2018MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separationcited by 19
  13. 2020Meta-learning Extractors for Music Source Separationcited by 18
  14. 2020Conditioned Source Separation for Music Instrument Performancescited by 18
  15. 2020Visually Guided Sound Source Separation using Cascaded Opponent Filter Networkcited by 15
  16. 2017Towards Estimating the Upper Bound of Visual-Speech Recognition: The Visual Lip-Reading Feasibility Databasecited by 13
  17. 2018Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environmentscited by 12
  18. 2020Deep Learning Based Source Separation Applied To Choir Ensemblescited by 9
  19. 2018End-to-End Sound Source Separation Conditioned On Instrument Labelscited by 5
  20. 2020Deep Variational Generative Models for Audio-visual Speech Separationcited by 4
  21. 2020Content based singing voice source separation via strong conditioning using aligned phonemescited by 2
  22. 2021Face-GCN: A Graph Convolutional Network for 3D Dynamic Face Identification/Recognitioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 3; 2025: 3; 2026: 2.

12 papers.

  1. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  2. 2022VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voicescited by 18
  3. 2025Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localizationcited by 17
  4. 2022VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformercited by 14
  5. 2022An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuitscited by 9
  6. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusioncited by 8
  7. 2025PianoVAM: A Multimodal Piano Performance Datasetcited by 5
  8. 2021Towards Robust Real-time Audio-Visual Speech Enhancementcited by 2
  9. 2024SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge Evaluation Plancited by 2
  10. 2024TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusioncited by 1
  11. 2026VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categoriescited by 1
  12. 2026MambaVoice: Lightweight Audiovisual Singing Voice Separation Via A Hybrid Mamba-Transformer Modelcited by 0