# The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio

arXiv: https://arxiv.org/abs/2405.04880 · HTML: https://mlsanity.com/paper/2405.04880/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 92 references that are in the map.
- First posted on arXiv 8 May 2024.

## Builds on (most cited first, 30 of 67)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)

## Built on by (most cited first, 30 of 31)

- [Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models](https://mlsanity.com/paper/2509.02859/) (2025)
- [Mixture of Experts Fusion for Fake Audio Detection Using Frozen wav2vec 2.0](https://mlsanity.com/paper/2409.11909/) (2024)
- [Nes2Net: A Lightweight Nested Architecture for Foundation Model Driven Speech Anti-spoofing](https://mlsanity.com/paper/2504.05657/) (2025)
- [Where are we in audio deepfake detection? A systematic analysis over generative and detection models](https://mlsanity.com/paper/2410.04324/) (2024)
- [Detect All-Type Deepfake Audio: Wavelet Prompt Tuning for Enhanced Auditory Perception](https://mlsanity.com/paper/2504.06753/) (2025)
- [Adversarial Attacks on Audio Deepfake Detection: A Benchmark and Comparative Study](https://mlsanity.com/paper/2509.07132/) (2025)
- [Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR](https://mlsanity.com/paper/2603.06164/) (2026)
- [Diversity Matters: Dataset Diversification and Dual-Branch Network for Generalized AI-Generated Image Detection](https://mlsanity.com/paper/2603.27800/) (2026)
- [Does Current Deepfake Audio Detection Model Effectively Detect ALM-based Deepfake Audio?](https://mlsanity.com/paper/2408.10853/) (2024)
- [CoReLoop: Parameter-Efficient Controlled Recurrent Refinement for Audio Deepfake Detection](https://mlsanity.com/paper/2609.19818/) (2026)
- [Temporal Variability and Multi-Viewed Self-Supervised Representations to Tackle the ASVspoof5 Deepfake Challenge](https://mlsanity.com/paper/2408.06922/) (2024)
- [Can DeepFake Speech be Reliably Detected?](https://mlsanity.com/paper/2410.06572/) (2024)
- [End-to-end Audio Deepfake Detection from RAW Waveforms: a RawNet-Based Approach with Cross-Dataset Evaluation](https://mlsanity.com/paper/2504.20923/) (2025)
- [Fake Speech Wild: Detecting Deepfake Speech on Social Media Platform](https://mlsanity.com/paper/2508.10559/) (2025)
- [On Deepfake Voice Detection -- It's All in the Presentation](https://mlsanity.com/paper/2509.26471/) (2025)
- [Alethia: A Foundational Encoder for Voice Deepfakes](https://mlsanity.com/paper/2605.00251/) (2026)
- [Teffic-Audio: Tell Fact from Fiction](https://mlsanity.com/paper/2607.28351/) (2026)
- [Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge System](https://mlsanity.com/paper/2508.20983/) (2025)
- [LJ-Spoof: A Generatively Varied Corpus for Audio Anti-Spoofing and Synthesis Source Tracing](https://mlsanity.com/paper/2601.07958/) (2026)
- [HyperPotter: Spell the Charm of High-Order Interactions in Audio Deepfake Detection](https://mlsanity.com/paper/2602.05670/) (2026)
- [A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection](https://mlsanity.com/paper/2603.01482/) (2026)
- [Spectro-Temporal Modulation Representation Framework for Human-Imitated Speech Detection](https://mlsanity.com/paper/2604.23241/) (2026)
- [Speaker-Invariant Representation Learning for Spoofing Detection via Gradient Reversal and A Variational Information Bottleneck](https://mlsanity.com/paper/2606.08678/) (2026)
- [From Self-Supervised Speech Models to Mixture-of-Experts for Robust Anti-Spoofing](https://mlsanity.com/paper/2606.14639/) (2026)
- [Linguistic Bias Mitigation for Spoofing Detection via Gradient Reversal and A Variational Information Bottleneck](https://mlsanity.com/paper/2606.31411/) (2026)
- [Time-Frequency Consistency Learning for Robust Speech Deepfake Detection](https://mlsanity.com/paper/2607.17761/) (2026)
- [Trajectory Dynamics in Self-Supervised Learning Latent Space for Audio Deepfake Detection](https://mlsanity.com/paper/2608.13817/) (2026)
- [ToolDF: Tool-Integrated Reasoning for Mixed-Authenticity Audio Deepfake Detection](https://mlsanity.com/paper/2609.03620/) (2026)
- [Robust Workflow Generation via Adversarial Learning for Audio Deepfake Detection](https://mlsanity.com/paper/2609.20063/) (2026)
- [What Survives the Codec Shift: Pooled No-Vocals Residuals for Speech Deepfake Detection](https://mlsanity.com/paper/2609.33375/) (2026)

## Abstract (arXiv)

With the proliferation of Audio Language Model (ALM) based deepfake audio, there is an urgent need for generalized detection methods. ALM-based deepfake audio currently exhibits widespread, high deception, and type versatility, posing a significant challenge to current audio deepfake detection (ADD) models trained solely on vocoded data. To effectively detect ALM-based deepfake audio, we focus on the mechanism of the ALM-based audio generation method, the conversion from neural codec to waveform. We initially constructed the Codecfake dataset, an open-source, large-scale collection comprising over 1 million audio samples in both English and Chinese, focus on ALM-based audio detection. As countermeasure, to achieve universal detection of deepfake audio and tackle domain ascent bias issue of original sharpness aware minimization (SAM), we propose the CSAM strategy to learn a domain balanced and generalized minima. In our experiments, we first demonstrate that ADD model training with the Codecfake dataset can effectively detects ALM-based audio. Furthermore, our proposed generalization countermeasure yields the lowest average equal error rate (EER) of 0.616% across all test conditions compared to baseline models. The dataset and associated code are available online.