# Are Music Foundation Models Better at Singing Voice Deepfake Detection? Far-Better Fuse them with Speech Foundation Models

arXiv: https://arxiv.org/abs/2409.14131 · HTML: https://mlsanity.com/paper/2409.14131/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 21 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Audio Deepfake Detection: A Survey](https://mlsanity.com/paper/2308.14970/) (2023)
- [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training](https://mlsanity.com/paper/2110.05752/) (2021)
- [Do Vision and Language Encoders Represent the World Similarly?](https://mlsanity.com/paper/2401.05224/) (2024)
- [HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis](https://mlsanity.com/paper/2009.01776/) (2020)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System](https://mlsanity.com/paper/2006.06261/) (2020)
- [DeepSinger: Singing Voice Synthesis with Data Mined From the Web](https://mlsanity.com/paper/2007.04590/) (2020)
- [The Singing Voice Conversion Challenge 2023](https://mlsanity.com/paper/2306.14422/) (2023)
- [Unsupervised Cross-Domain Singing Voice Conversion](https://mlsanity.com/paper/2008.02830/) (2020)
- [MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning](https://mlsanity.com/paper/2212.02508/) (2022)
- [SingFake: Singing Voice Deepfake Detection](https://mlsanity.com/paper/2309.07525/) (2023)
- [SAMO: Speaker Attractor Multi-Center One-Class Learning for Voice Anti-Spoofing](https://mlsanity.com/paper/2211.02718/) (2022)
- [CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detection](https://mlsanity.com/paper/2406.02438/) (2024)
- [PitchNet: Unsupervised Singing Voice Conversion with Pitch Adversarial Network](https://mlsanity.com/paper/1912.01852/) (2019)
- [Heterogeneity over Homogeneity: Investigating Multilingual Speech Pre-Trained Models for Detecting Audio Deepfake](https://mlsanity.com/paper/2404.00809/) (2024)
- [Learning From Yourself: A Self-Distillation Method for Fake Speech Detection](https://mlsanity.com/paper/2303.01211/) (2023)
- [FSD: An Initial Chinese Dataset for Fake Song Detection](https://mlsanity.com/paper/2309.02232/) (2023)
- [Singing Voice Graph Modeling for SingFake Detection](https://mlsanity.com/paper/2406.03111/) (2024)
- [Self-Supervised Representations for Singing Voice Conversion](https://mlsanity.com/paper/2303.12197/) (2023)
- [Transforming the Embeddings: A Lightweight Technique for Speech Emotion Recognition Tasks](https://mlsanity.com/paper/2305.18640/) (2023)

## Abstract (arXiv)

In this study, for the first time, we extensively investigate whether music foundation models (MFMs) or speech foundation models (SFMs) work better for singing voice deepfake detection (SVDD), which has recently attracted attention in the research community. For this, we perform a comprehensive comparative study of state-of-the-art (SOTA) MFMs (MERT variants and music2vec) and SFMs (pre-trained for general speech representation learning as well as speaker recognition). We show that speaker recognition SFM representations perform the best amongst all the foundation models (FMs), and this performance can be attributed to its higher efficacy in capturing the pitch, tone, intensity, etc, characteristics present in singing voices. To our end, we also explore the fusion of FMs for exploiting their complementary behavior for improved SVDD, and we propose a novel framework, FIONA for the same. With FIONA, through the synchronization of x-vector (speaker recognition SFM) and MERT-v1-330M (MFM), we report the best performance with the lowest Equal Error Rate (EER) of 13.74 %, beating all the individual FMs as well as baseline FM fusions and achieving SOTA results.