# What to Remember: Self-Adaptive Continual Learning for Audio Deepfake Detection

arXiv: https://arxiv.org/abs/2312.09651 · HTML: https://mlsanity.com/paper/2312.09651/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection](https://mlsanity.com/paper/2109.00537/) (2021)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [One-class Learning Towards Synthetic Voice Spoofing Detection](https://mlsanity.com/paper/2010.13995/) (2020)
- [Investigating self-supervised front ends for speech spoofing countermeasures](https://mlsanity.com/paper/2111.07725/) (2021)
- [The Vicomtech Audio Deepfake Detection System based on Wav2Vec2 for the 2022 ADD Challenge](https://mlsanity.com/paper/2203.01573/) (2022)
- [Continual Learning for Fake Audio Detection](https://mlsanity.com/paper/2104.07286/) (2021)
- [An Empirical Study on Channel Effects for Synthetic Voice Spoofing Countermeasure Systems](https://mlsanity.com/paper/2104.01320/) (2021)

## Built on by (most cited first, 15 of 15)

- [GenVidBench: A 6-Million Benchmark for AI-Generated Video Detection](https://mlsanity.com/paper/2501.11340/) (2025)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [CLAD: Robust Audio Deepfake Detection Against Manipulation Attacks with Contrastive Learning](https://mlsanity.com/paper/2404.15854/) (2024)
- [VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing](https://mlsanity.com/paper/2404.06674/) (2024)
- [VoiceWukong: Benchmarking Deepfake Voice Detection](https://mlsanity.com/paper/2409.06348/) (2024)
- [Region-Based Optimization in Continual Learning for Audio Deepfake Detection](https://mlsanity.com/paper/2412.11551/) (2024)
- [Passive Deepfake Detection Across Multi-modalities: A Comprehensive Survey](https://mlsanity.com/paper/2411.17911/) (2024)
- [Characterizing Continual Learning Scenarios and Strategies for Audio Analysis](https://mlsanity.com/paper/2407.00465/) (2024)
- [Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processes](https://mlsanity.com/paper/2505.23619/) (2025)
- [An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio](https://mlsanity.com/paper/2407.08239/) (2024)
- [Rehearsal with Auxiliary-Informed Sampling for Audio Deepfake Detection](https://mlsanity.com/paper/2505.24486/) (2025)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [Zero-Shot to Zero-Lies: Detecting Bengali Deepfake Audio through Transfer Learning](https://mlsanity.com/paper/2512.21702/) (2025)
- [Traceback Translators Against Forgetting in Continual Fake Speech Detection](https://mlsanity.com/paper/2607.12569/) (2026)
- [Learning as Deepfakes Evolve: RF-Prompt for Continual Audio Deepfake Detection](https://mlsanity.com/paper/2609.37586/) (2026)

## Abstract (arXiv)

The rapid evolution of speech synthesis and voice conversion has raised substantial concerns due to the potential misuse of such technology, prompting a pressing need for effective audio deepfake detection mechanisms. Existing detection models have shown remarkable success in discriminating known deepfake audio, but struggle when encountering new attack types. To address this challenge, one of the emergent effective approaches is continual learning. In this paper, we propose a continual learning approach called Radian Weight Modification (RWM) for audio deepfake detection. The fundamental concept underlying RWM involves categorizing all classes into two groups: those with compact feature distributions across tasks, such as genuine audio, and those with more spread-out distributions, like various types of fake audio. These distinctions are quantified by means of the in-class cosine distance, which subsequently serves as the basis for RWM to introduce a trainable gradient modification direction for distinct data types. Experimental evaluations against mainstream continual learning methods reveal the superiority of RWM in terms of knowledge acquisition and mitigating forgetting in audio deepfake detection. Furthermore, RWM's applicability extends beyond audio deepfake detection, demonstrating its potential significance in diverse machine learning domains such as image recognition.