# When Fine-Tuning is Not Enough: Lessons from HSAD on Hybrid and Adversarial Audio Spoof Detection

arXiv: https://arxiv.org/abs/2509.07323 · HTML: https://mlsanity.com/paper/2509.07323/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Anti-spoofing and adversarial audio attacks, one of 11 directions in Speech and audio language models.
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- Speech and audio language models / [Anti-spoofing and adversarial audio attacks](https://mlsanity.com/speech/anti-spoofing-and-adversarial-audio-attacks/)

## Builds on (most cited first, 11 of 11)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [ADD 2023: the Second Audio Deepfake Detection Challenge](https://mlsanity.com/paper/2305.13774/) (2023)
- [ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech](https://mlsanity.com/paper/2102.05889/) (2021)
- [ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiency](https://mlsanity.com/paper/2406.02167/) (2024)
- [Optimizing Tandem Speaker Verification and Anti-Spoofing Systems](https://mlsanity.com/paper/2201.09709/) (2022)
- [Vision Transformer: Vit and its Derivatives](https://mlsanity.com/paper/2205.11239/) (2022)
- [A Comparison of Differential Performance Metrics for the Evaluation of Automatic Speaker Verification Fairness](https://mlsanity.com/paper/2404.17810/) (2024)
- [Two-Path GMM-ResNet and GMM-SENet for ASV Spoofing Detection](https://mlsanity.com/paper/2407.05605/) (2024)

## Abstract (arXiv)

The rapid advancement of AI has enabled highly realistic speech synthesis and voice cloning, posing serious risks to voice authentication, smart assistants, and telecom security. While most prior work frames spoof detection as a binary task, real-world attacks often involve hybrid utterances that mix genuine and synthetic speech, making detection substantially more challenging. To address this gap, we introduce the Hybrid Spoofed Audio Dataset (HSAD), a benchmark containing 1,248 clean and 41,044 degraded utterances across four classes: human, cloned, zero-shot AI-generated, and hybrid audio. Each sample is annotated with spoofing method, speaker identity, and degradation metadata to enable fine-grained analysis. We evaluate six transformer-based models, including spectrogram encoders (MIT-AST, MattyB95-AST) and self-supervised waveform models (Wav2Vec2, HuBERT). Results reveal critical lessons: pretrained models overgeneralize and collapse under hybrid conditions; spoof-specific fine-tuning improves separability but struggles with unseen compositions; and dataset-specific adaptation on HSAD yields large performance gains (AST greater than 97 percent and F1 score is approximately 99 percent), though residual errors persist for complex hybrids. These findings demonstrate that fine-tuning alone is not sufficient-robust hybrid-aware benchmarks like HSAD are essential to expose calibration failures, model biases, and factors affecting spoof detection in adversarial environments. HSAD thus provides both a dataset and an analytic framework for building resilient and trustworthy voice authentication systems.