# Parameter-Efficient Transfer Learning of Audio Spectrogram Transformers

arXiv: https://arxiv.org/abs/2312.03694 · HTML: https://mlsanity.com/paper/2312.03694/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2023.

## Built on by (most cited first, 8 of 8)

- [Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach](https://mlsanity.com/paper/2505.14336/) (2025)
- [Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters](https://mlsanity.com/paper/2402.00828/) (2024)
- [MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2510.04136/) (2025)
- [Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation](https://mlsanity.com/paper/2511.16757/) (2025)
- [Transfer Learning in Vocal Education: Technical Evaluation of Limited Samples Describing Mezzo-soprano](https://mlsanity.com/paper/2410.23325/) (2024)
- [TAIL: Text-Audio Incremental Learning](https://mlsanity.com/paper/2503.04258/) (2025)
- [DeepForestSound: a multi-species automatic detector for passive acoustic monitoring in African tropical forests, a case study in Kibale National Park](https://mlsanity.com/paper/2604.08087/) (2026)
- [MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio](https://mlsanity.com/paper/2606.15638/) (2026)

## Abstract (arXiv)

Parameter-efficient transfer learning (PETL) methods have emerged as a solid alternative to the standard full fine-tuning approach. They only train a few extra parameters for each downstream task, without sacrificing performance and dispensing with the issue of storing a copy of the pre-trained model for each task. For audio classification tasks, the Audio Spectrogram Transformer (AST) model shows impressive results. However, surprisingly, how to efficiently adapt it to several downstream tasks has not been tackled before. In this paper, we bridge this gap and present a detailed investigation of common PETL methods for the adaptation of the AST model to audio/speech tasks. Furthermore, we propose a new adapter design that exploits the convolution module of the Conformer model, leading to superior performance over the standard PETL approaches and surpassing or achieving performance parity with full fine-tuning by updating only 0.29% of the parameters. Finally, we provide ablation studies revealing that our proposed adapter: 1) proves to be effective in few-shot efficient transfer learning, 2) attains optimal results regardless of the amount of the allocated parameters, and 3) can be applied to other pre-trained models.