# Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio

arXiv: https://arxiv.org/abs/2406.08112 · HTML: https://mlsanity.com/paper/2406.08112/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [Does Audio Deepfake Detection Generalize?](https://mlsanity.com/paper/2203.16263/) (2022)
- [UniAudio: An Audio Foundation Model Toward Universal Audio Generation](https://mlsanity.com/paper/2310.00704/) (2023)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec](https://mlsanity.com/paper/2305.02765/) (2023)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation](https://mlsanity.com/paper/2305.16107/) (2023)
- [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](https://mlsanity.com/paper/2111.02813/) (2021)
- [LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT](https://mlsanity.com/paper/2310.04673/) (2023)
- [SpeechX: Neural Codec Language Model as a Versatile Speech Transformer](https://mlsanity.com/paper/2308.06873/) (2023)
- [ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech](https://mlsanity.com/paper/2102.05889/) (2021)
- [STC Antispoofing Systems for the ASVspoof2019 Challenge](https://mlsanity.com/paper/1904.05576/) (2019)
- [AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec](https://mlsanity.com/paper/2305.16608/) (2023)
- [FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec](https://mlsanity.com/paper/2309.07405/) (2023)
- [AI-Synthesized Voice Detection Using Neural Vocoder Artifacts](https://mlsanity.com/paper/2304.13085/) (2023)
- [HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks](https://mlsanity.com/paper/2006.05694/) (2020)
- [The Vicomtech Audio Deepfake Detection System based on Wav2Vec2 for the 2022 ADD Challenge](https://mlsanity.com/paper/2203.01573/) (2022)
- [Can large-scale vocoded spoofed data improve speech spoofing countermeasure with a self-supervised front end?](https://mlsanity.com/paper/2309.06014/) (2023)

## Built on by (most cited first, 6 of 6)

- [Lightweight Model Attribution and Detection of Synthetic Speech via Audio Residual Fingerprints](https://mlsanity.com/paper/2411.14013/) (2024)
- [PhonemeFake: Redefining Deepfake Realism with Language-Driven Segmental Manipulation and Adaptive Bilevel Detection](https://mlsanity.com/paper/2506.22783/) (2025)
- [Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensics](https://mlsanity.com/paper/2508.00784/) (2025)
- [Ethical and Technical Limits of Deepfake Speech Datasets](https://mlsanity.com/paper/2606.10911/) (2026)
- [Audio Deepfake Detection in the Age of Advanced Text-to-Speech models](https://mlsanity.com/paper/2601.20510/) (2026)
- [An Initial Study of Physiological Markers for Deepfake Speech Detection](https://mlsanity.com/paper/2610.18934/) (2026)

## Abstract (arXiv)

With the proliferation of Large Language Model (LLM) based deepfake audio, there is an urgent need for effective detection methods. Previous deepfake audio generation methods typically involve a multi-step generation process, with the final step using a vocoder to predict the waveform from handcrafted features. However, LLM-based audio is directly generated from discrete neural codecs in an end-to-end generation process, skipping the final step of vocoder processing. This poses a significant challenge for current audio deepfake detection (ADD) models based on vocoder artifacts. To effectively detect LLM-based deepfake audio, we focus on the core of the generation process, the conversion from neural codec to waveform. We propose Codecfake dataset, which is generated by seven representative neural codec methods. Experiment results show that codec-trained ADD models exhibit a 41.406% reduction in average equal error rate compared to vocoder-trained ADD models on the Codecfake test set.