# Adapter Incremental Continual Learning of Efficient Audio Spectrogram Transformers

arXiv: https://arxiv.org/abs/2302.14314 · HTML: https://mlsanity.com/paper/2302.14314/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2023.

## Built on by (most cited first, 5 of 5)

- [Characterizing Continual Learning Scenarios and Strategies for Audio Analysis](https://mlsanity.com/paper/2407.00465/) (2024)
- [ISCUTE: Instance Segmentation of Cables Using Text Embedding](https://mlsanity.com/paper/2402.11996/) (2024)
- [UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation](https://mlsanity.com/paper/2602.21772/) (2026)
- [Audio Contrastive-based Fine-tuning: Decoupling Representation Learning and Classification](https://mlsanity.com/paper/2309.11895/) (2023)
- [TAIL: Text-Audio Incremental Learning](https://mlsanity.com/paper/2503.04258/) (2025)

## Abstract (arXiv)

Continual learning involves training neural networks incrementally for new tasks while retaining the knowledge of previous tasks. However, efficiently fine-tuning the model for sequential tasks with minimal computational resources remains a challenge. In this paper, we propose Task Incremental Continual Learning (TI-CL) of audio classifiers with both parameter-efficient and compute-efficient Audio Spectrogram Transformers (AST). To reduce the trainable parameters without performance degradation for TI-CL, we compare several Parameter Efficient Transfer (PET) methods and propose AST with Convolutional Adapters for TI-CL, which has less than 5% of trainable parameters of the fully fine-tuned counterparts. To reduce the computational complexity, we introduce a novel Frequency-Time factorized Attention (FTA) method that replaces the traditional self-attention in transformers for audio spectrograms. FTA achieves competitive performance with only a factor of the computations required by Global Self-Attention (GSA). Finally, we formulate our method for TI-CL, called Adapter Incremental Continual Learning (AI-CL), as a combination of the "parameter-efficient" Convolutional Adapter and the "compute-efficient" FTA. Experiments on ESC-50, SpeechCommandsV2 (SCv2), and Audio-Visual Event (AVE) benchmarks show that our proposed method prevents catastrophic forgetting in TI-CL while maintaining a lower computational budget.