# Quantizer-Aware Hierarchical Neural Codec Modeling for Speech Deepfake Detection

arXiv: https://arxiv.org/abs/2603.16914 · HTML: https://mlsanity.com/paper/2603.16914/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 25 of 25)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [Does Audio Deepfake Detection Generalize?](https://mlsanity.com/paper/2203.16263/) (2022)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scale](https://mlsanity.com/paper/2408.08739/) (2024)
- [ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech](https://mlsanity.com/paper/2102.05889/) (2021)
- [Investigating self-supervised front ends for speech spoofing countermeasures](https://mlsanity.com/paper/2111.07725/) (2021)
- [SafeEar: Content Privacy-Preserving Audio Deepfake Detection](https://mlsanity.com/paper/2409.09272/) (2024)
- [A robust audio deepfake detection system via multi-view feature](https://mlsanity.com/paper/2403.01960/) (2024)
- [Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detection](https://mlsanity.com/paper/2406.10283/) (2024)
- [CodecFake: Enhancing Anti-Spoofing Models Against Deepfake Audios from Codec-Based Speech Synthesis Systems](https://mlsanity.com/paper/2406.07237/) (2024)
- [WavLM model ensemble for audio deepfake detection](https://mlsanity.com/paper/2408.07414/) (2024)
- [Fully Automated End-to-End Fake Audio Detection](https://mlsanity.com/paper/2208.09618/) (2022)
- [Speech Foundation Model Ensembles for the Controlled Singing Voice Deepfake Detection (CtrSVDD) Challenge 2024](https://mlsanity.com/paper/2409.02302/) (2024)
- [Codec-Based Deepfake Source Tracing via Neural Audio Codec Taxonomy](https://mlsanity.com/paper/2505.12994/) (2025)
- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [MoLEx: Mixture of LoRA Experts in Speech Self-Supervised Models for Audio Deepfake Detection](https://mlsanity.com/paper/2509.09175/) (2025)
- [WaveSP-Net: Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detection](https://mlsanity.com/paper/2510.05305/) (2025)
- [Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs](https://mlsanity.com/paper/2511.16639/) (2025)
- [Two Views, One Truth: Spectral and Self-Supervised Features Fusion for Robust Speech Deepfake Detection](https://mlsanity.com/paper/2507.20417/) (2025)

## Abstract (arXiv)

Neural audio codecs discretize speech via residual vector quantization (RVQ), forming a coarse-to-fine hierarchy across quantizers. While codec models have been explored for representation learning, their discrete structure remains underutilized in speech deepfake detection. In particular, different quantization levels capture complementary acoustic cues, where early quantizers encode coarse structure and later quantizers refine residual details that reveal synthesis artifacts. Existing systems either rely on continuous encoder features or ignore this quantizer-level hierarchy. We propose a hierarchy-aware representation learning framework that models quantizer-level contributions through learnable global weighting, enabling structured codec representations aligned with forensic cues. Keeping the speech encoder backbone frozen and updating only 4.4% additional parameters, our method achieves relative EER reductions of 46.2% on ASVspoof 2019 and 13.9% on ASVspoof5 over strong baselines.