# Not All Attacks Are Learned Equally in Speech Deepfake Detection

arXiv: https://arxiv.org/abs/2609.11763 · HTML: https://mlsanity.com/paper/2609.11763/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Anti-spoofing and adversarial audio attacks, one of 11 directions in Speech and audio language models.
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Placed in

- Speech and audio language models / [Anti-spoofing and adversarial audio attacks](https://mlsanity.com/speech/anti-spoofing-and-adversarial-audio-attacks/)

## Builds on (most cited first, 30 of 35)

- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [End-to-end anti-spoofing with RawNet2](https://mlsanity.com/paper/2011.01108/) (2020)
- [Does Audio Deepfake Detection Generalize?](https://mlsanity.com/paper/2203.16263/) (2022)
- [ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild](https://mlsanity.com/paper/2210.02437/) (2022)
- [Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation](https://mlsanity.com/paper/2202.12233/) (2022)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scale](https://mlsanity.com/paper/2408.08739/) (2024)
- [ADD 2023: the Second Audio Deepfake Detection Challenge](https://mlsanity.com/paper/2305.13774/) (2023)
- [MLAAD: The Multi-Language Audio Anti-Spoofing Dataset](https://mlsanity.com/paper/2401.09512/) (2024)
- [XLSR-Mamba: A Dual-Column Bidirectional State Space Model for Spoofing Attack Detection](https://mlsanity.com/paper/2411.10027/) (2024)
- [A Survey on Speech Deepfake Detection](https://mlsanity.com/paper/2404.13914/) (2024)
- [SpoofCeleb: Speech Deepfake Detection and SASV In The Wild](https://mlsanity.com/paper/2409.17285/) (2024)
- [ASSERT: Anti-Spoofing with Squeeze-Excitation and Residual neTworks](https://mlsanity.com/paper/1904.01120/) (2019)
- [Light Convolutional Neural Network with Feature Genuinization for Detection of Synthetic Speech Attacks](https://mlsanity.com/paper/2009.09637/) (2020)
- [ALLM4ADD: Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detection](https://mlsanity.com/paper/2505.11079/) (2025)
- [What to Remember: Self-Adaptive Continual Learning for Audio Deepfake Detection](https://mlsanity.com/paper/2312.09651/) (2023)
- [EmoFake: An Initial Dataset for Emotion Fake Audio Detection](https://mlsanity.com/paper/2211.05363/) (2022)
- [Detection of Cross-Dataset Fake Audio Based on Prosodic and Pronunciation Features](https://mlsanity.com/paper/2305.13700/) (2023)
- [Attacker Attribution of Audio Deepfakes](https://mlsanity.com/paper/2203.15563/) (2022)
- [Multilingual Source Tracing of Speech Deepfakes: A First Benchmark](https://mlsanity.com/paper/2508.04143/) (2025)
- [Neural Codec Source Tracing: Toward Comprehensive Attribution in Open-Set Condition](https://mlsanity.com/paper/2501.06514/) (2025)
- [DFALLM: Achieving Generalizable Multitask Deepfake Detection by Optimizing Audio LLM Components](https://mlsanity.com/paper/2512.08403/) (2025)
- [Region-Based Optimization in Continual Learning for Audio Deepfake Detection](https://mlsanity.com/paper/2412.11551/) (2024)
- [Towards Explicit Acoustic Evidence Perception in Audio LLMs for Speech Deepfake Detection](https://mlsanity.com/paper/2601.23066/) (2026)
- [Can Emotion Fool Anti-spoofing?](https://mlsanity.com/paper/2505.23962/) (2025)
- [Towards Robust Speech Deepfake Detection via Human-Inspired Reasoning](https://mlsanity.com/paper/2603.10725/) (2026)
- [ProSDD: Learning Prosodic Representations for Speech Deepfake Detection against Expressive and Emotional Attacks](https://mlsanity.com/paper/2604.13229/) (2026)
- [Naturalness-Aware Curriculum Learning with Dynamic Temperature for Speech Deepfake Detection](https://mlsanity.com/paper/2505.13976/) (2025)
- [Fake Speech Wild: Detecting Deepfake Speech on Social Media Platform](https://mlsanity.com/paper/2508.10559/) (2025)
- [ESDD 2026: Environmental Sound Deepfake Detection Challenge Evaluation Plan](https://mlsanity.com/paper/2508.04529/) (2025)

## Abstract (arXiv)

Speech deepfake detection (SDD) models are trained on multi-attack datasets containing diverse spoofing systems, such as text-to-speech (TTS) and voice conversion (VC). In standard classifier training on multi-attack datasets, all attacks are treated as one spoofed class, and performance is reported using overall Equal Error Rate (EER). This aggregate view obscures how individual attacks shape learning and generalization. To better understand this attack-level behavior, we first balance TTS and VC exposure using sample and attack omission. We then measure attack-wise EER at inference and analyze attack-wise training loss and predictive entropy to characterize optimization. Results show that attacks contribute unequally: some attacks have high EER sensitivity and concentrated entropy with low loss, indicating strong influence on the decision boundary. We define these as high-impact attacks. To reduce uneven generalization across attacks, we propose a replay-regularized, attack-aware curriculum that steps exposure based on measured attack influence. Experiments on ASVspoof 2019, 2021, ASVspoof 5, and Fake-or-Real show improved overall robustness and reduced attack-level imbalance compared with standard multi-attack training.