Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learning
Not on the map. Why?
Abstract (arXiv)
The rapid evolution of end-to-end AI music generation poses an escalating threat to artistic authenticity and copyright, demanding detection methods that can keep pace. While foundational, existing models like SpecTTTra falter when faced with the diverse and rapidly advancing ecosystem of new generators, exhibiting significant performance drops on out-of-distribution (OOD) content. This generalization failure highlights a critical gap: the need for more challenging benchmarks and more robust detection architectures. To address this, we first introduce Melody or Machine (MoM), a new large-scale benchmark of over 130,000 songs (6,665 hours). MoM is the most diverse dataset to date, built with a mix of open and closed-source models and a curated OOD test set designed specifically to foster the development of truly generalizable detectors. Alongside this benchmark, we introduce CLAM, a novel dual-stream detection architecture. We hypothesize that subtle, machine-induced inconsistencies between vocal and instrumental elements, often imperceptible in a mixed signal, offer a powerful tell-tale sign of synthesis. CLAM is designed to test this hypothesis by employing two distinct pre-trained audio encoders (MERT and Wave2Vec2) to create parallel representations of the audio. These representations are fused by a learnable cross-aggregation module that models their inter-dependencies. The model is trained with a dual-loss objective: a standard binary cross-entropy loss for classification, complemented by a contrastive triplet loss which trains the model to distinguish between coherent and artificially mismatched stream pairings, enhancing its sensitivity to synthetic artifacts without presuming a simple feature alignment. CLAM establishes a new state-of-the-art in synthetic music forensics. It achieves an F1 score of 0.925 on our challenging MoM benchmark.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 29 Nov 2025.
Builds on
24 of 34 references.
References · 24
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2015An Introduction to Convolutional Neural Networkscited by 510
- 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
- 2016FMA: A Dataset For Music Analysiscited by 205
- 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
- 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusioncited by 46
- 2023Audio Deepfake Detection with Self-Supervised WavLM and Multi-Fusion Attentive Classifiercited by 26
- 2023SingFake: Singing Voice Deepfake Detectioncited by 21
- 2024SONICS: Synthetic Or Not -- Identifying Counterfeit Songscited by 21
12 more
- 2024CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detectioncited by 16
- 2024Heterogeneity over Homogeneity: Investigating Multilingual Speech Pre-Trained Models for Detecting Audio Deepfakecited by 14
- 2020Expertise and Dynamics within Crowdsourced Musical Knowledge Curation: A Case Study of the Genius Platformcited by 13
- 2024X-LLaVA: Optimizing Bilingual Large Vision-Language Alignmentcited by 10
- 2018Learning to Recognize Musical Genre from Audiocited by 9
- 2021RefineGAN: Universally Generating Waveform Better than Ground Truth with Highly Accurate Pitch and Intensity Responsescited by 5
- 2024Deepfake Audio Detection Using Spectrogram-based Feature and Ensemble of Deep Learning Modelscited by 5
- 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Imagescited by 5
- 2021MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesiscited by 4
- 2024Investigating Prosodic Signatures via Speech Pre-Trained Models for Audio Deepfake Source Attributioncited by 4
- 2025Generalizable Audio Deepfake Detection via Hierarchical Structure Learning and Feature Whitening in Poincar'e spherecited by 4
- 2025Deepfake Detection of Singing Voices With Whisper Encodingscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.