# Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detection

arXiv: https://arxiv.org/abs/2308.03300 · HTML: https://mlsanity.com/paper/2308.03300/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 7 Aug 2023.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection](https://mlsanity.com/paper/2109.00537/) (2021)
- [Does Audio Deepfake Detection Generalize?](https://mlsanity.com/paper/2203.16263/) (2022)
- [Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation](https://mlsanity.com/paper/2202.12233/) (2022)
- [Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild](https://mlsanity.com/paper/1903.12648/) (2019)
- [Incremental Few-Shot Object Detection](https://mlsanity.com/paper/2003.04668/) (2020)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)

## Built on by (most cited first, 11 of 11)

- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [CLAD: Robust Audio Deepfake Detection Against Manipulation Attacks with Contrastive Learning](https://mlsanity.com/paper/2404.15854/) (2024)
- [Prompt Tuning for Audio Deepfake Detection: Computationally Efficient Test-time Domain Adaptation with Limited Target Dataset](https://mlsanity.com/paper/2410.09869/) (2024)
- [Continual Audio-Visual Sound Separation](https://mlsanity.com/paper/2411.02860/) (2024)
- [Improving Generalization for AI-Synthesized Voice Detection](https://mlsanity.com/paper/2412.19279/) (2024)
- [Region-Based Optimization in Continual Learning for Audio Deepfake Detection](https://mlsanity.com/paper/2412.11551/) (2024)
- [Seamless Language Expansion: Enhancing Multilingual Mastery in Self-Supervised Models](https://mlsanity.com/paper/2406.14092/) (2024)
- [Torque-Aware Momentum](https://mlsanity.com/paper/2412.18790/) (2024)
- [An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio](https://mlsanity.com/paper/2407.08239/) (2024)
- [Rehearsal with Auxiliary-Informed Sampling for Audio Deepfake Detection](https://mlsanity.com/paper/2505.24486/) (2025)
- [Learning as Deepfakes Evolve: RF-Prompt for Continual Audio Deepfake Detection](https://mlsanity.com/paper/2609.37586/) (2026)

## Abstract (arXiv)

Current fake audio detection algorithms have achieved promising performances on most datasets. However, their performance may be significantly degraded when dealing with audio of a different dataset. The orthogonal weight modification to overcome catastrophic forgetting does not consider the similarity of genuine audio across different datasets. To overcome this limitation, we propose a continual learning algorithm for fake audio detection to overcome catastrophic forgetting, called Regularized Adaptive Weight Modification (RAWM). When fine-tuning a detection network, our approach adaptively computes the direction of weight modification according to the ratio of genuine utterances and fake utterances. The adaptive modification direction ensures the network can effectively detect fake audio on the new dataset while preserving its knowledge of old model, thus mitigating catastrophic forgetting. In addition, genuine audio collected from quite different acoustic conditions may skew their feature distribution, so we introduce a regularization constraint to force the network to remember the old distribution in this regard. Our method can easily be generalized to related fields, like speech emotion recognition. We also evaluate our approach across multiple datasets and obtain a significant performance improvement on cross-dataset experiments.