# Deepfake Detection of Singing Voices With Whisper Encodings

arXiv: https://arxiv.org/abs/2501.18919 · HTML: https://mlsanity.com/paper/2501.18919/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 31 Jan 2025.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scale](https://mlsanity.com/paper/2408.08739/) (2024)
- [Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers](https://mlsanity.com/paper/2307.03183/) (2023)
- [Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed](https://mlsanity.com/paper/1909.01174/) (2019)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2201.08930/) (2022)
- [SingFake: Singing Voice Deepfake Detection](https://mlsanity.com/paper/2309.07525/) (2023)
- [Invariant Representations for Noisy Speech Recognition](https://mlsanity.com/paper/1612.01928/) (2016)
- [Learning Noise-Invariant Representations for Robust Speech Recognition](https://mlsanity.com/paper/1807.06610/) (2018)

## Built on by (most cited first, 2 of 2)

- [Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learning](https://mlsanity.com/paper/2512.00621/) (2025)
- [Beyond Artifacts: Towards Generalizable Synthetic Song Detection via Music-Intrinsic Features](https://mlsanity.com/paper/2606.16612/) (2026)

## Abstract (arXiv)

The deepfake generation of singing vocals is a concerning issue for artists in the music industry. In this work, we propose a singing voice deepfake detection (SVDD) system, which uses noise-variant encodings of open-AI's Whisper model. As counter-intuitive as it may sound, even though the Whisper model is known to be noise-robust, the encodings are rich in non-speech information, and are noise-variant. This leads us to evaluate Whisper encodings as feature representations for the SVDD task. Therefore, in this work, the SVDD task is performed on vocals and mixtures, and the performance is evaluated in %EER over varying Whisper model sizes and two classifiers- CNN and ResNet34, under different testing conditions.