Deepfake Detection of Singing Voices With Whisper Encodings
Not on the map. Why?
Abstract (arXiv)
The deepfake generation of singing vocals is a concerning issue for artists in the music industry. In this work, we propose a singing voice deepfake detection (SVDD) system, which uses noise-variant encodings of open-AI's Whisper model. As counter-intuitive as it may sound, even though the Whisper model is known to be noise-robust, the encodings are rich in non-speech information, and are noise-variant. This leads us to evaluate Whisper encodings as feature representations for the SVDD task. Therefore, in this work, the SVDD task is performed on vocals and mixtures, and the performance is evaluated in %EER over varying Whisper model sizes and two classifiers- CNN and ResNet34, under different testing conditions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 31 Jan 2025.
Builds on
12 of 17 references.
References · 12
- 2017Attention Is All You Needcited by 60,361
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2019pyannote.audio: neural building blocks for speaker diarizationcited by 181
- 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
- 2024ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scalecited by 91
- 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
- 2019Demucs: Deep Extractor for Music Sources with extra unlabeled data remixedcited by 54
- 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
- 2022A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognitioncited by 22
- 2023SingFake: Singing Voice Deepfake Detectioncited by 21
- 2016Invariant Representations for Noisy Speech Recognitioncited by 12
- 2018Learning Noise-Invariant Representations for Robust Speech Recognitioncited by 10
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.