mlsanityData to 7 Oct 2026

Are Music Foundation Models Better at Singing Voice Deepfake Detection? Far-Better Fuse them with Speech Foundation Models

arXiv 2409.14131 ↗First posted 21 Sep 2024eess.AS cs.LG cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

In this study, for the first time, we extensively investigate whether music foundation models (MFMs) or speech foundation models (SFMs) work better for singing voice deepfake detection (SVDD), which has recently attracted attention in the research community. For this, we perform a comprehensive comparative study of state-of-the-art (SOTA) MFMs (MERT variants and music2vec) and SFMs (pre-trained for general speech representation learning as well as speaker recognition). We show that speaker recognition SFM representations perform the best amongst all the foundation models (FMs), and this performance can be attributed to its higher efficacy in capturing the pitch, tone, intensity, etc, characteristics present in singing voices. To our end, we also explore the fusion of FMs for exploiting their complementary behavior for improved SVDD, and we propose a novel framework, FIONA for the same. With FIONA, through the synchronization of x-vector (speaker recognition SFM) and MERT-v1-330M (MFM), we report the best performance with the lowest Equal Error Rate (EER) of 13.74 %, beating all the individual FMs as well as baseline FM fusions and achieving SOTA results.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 35 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

31 of 35 references.

References · 31

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  5. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  6. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  7. 2021AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networkscited by 173
  8. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  9. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  10. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  11. 2023Audio Deepfake Detection: A Surveycited by 49
  12. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
19 more
  1. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 43
  2. 2020HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesiscited by 39
  3. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  4. 2020XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis Systemcited by 33
  5. 2020DeepSinger: Singing Voice Synthesis with Data Mined From the Webcited by 32
  6. 2023The Singing Voice Conversion Challenge 2023cited by 25
  7. 2020Unsupervised Cross-Domain Singing Voice Conversioncited by 23
  8. 2022MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learningcited by 22
  9. 2023SingFake: Singing Voice Deepfake Detectioncited by 21
  10. 2022SAMO: Speaker Attractor Multi-Center One-Class Learning for Voice Anti-Spoofingcited by 18
  11. 2024CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detectioncited by 16
  12. 2019PitchNet: Unsupervised Singing Voice Conversion with Pitch Adversarial Networkcited by 15
  13. 2024Heterogeneity over Homogeneity: Investigating Multilingual Speech Pre-Trained Models for Detecting Audio Deepfakecited by 14
  14. 2023FSD: An Initial Chinese Dataset for Fake Song Detectioncited by 12
  15. 2023Learning From Yourself: A Self-Distillation Method for Fake Speech Detectioncited by 9
  16. 2024Singing Voice Graph Modeling for SingFake Detectioncited by 9
  17. 2023Self-Supervised Representations for Singing Voice Conversioncited by 8
  18. 2023Transforming the Embeddings: A Lightweight Technique for Speech Emotion Recognition Taskscited by 4
  19. 2024Experimental Study: Enhancing Voice Spoofing Detection Models with wav2vec 2.0cited by 4

Built on by

Not cited yet.