# CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models

arXiv: https://arxiv.org/abs/2212.01282 · HTML: https://mlsanity.com/paper/2212.01282/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 20 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Efficient parametrization of multi-domain deep neural networks](https://mlsanity.com/paper/1803.10082/) (2018)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.03218/) (2022)
- [AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks](https://mlsanity.com/paper/2205.00305/) (2022)
- [Exploring Efficient-tuning Methods in Self-supervised Speech Models](https://mlsanity.com/paper/2210.06175/) (2022)
- [An Adapter Based Pre-Training for Efficient and Scalable Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2107.13530/) (2021)

## Built on by (most cited first, 13 of 13)

- [Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition](https://mlsanity.com/paper/2310.06434/) (2023)
- [Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition](https://mlsanity.com/paper/2309.15223/) (2023)
- [How to Estimate Model Transferability of Pre-Trained Speech Models?](https://mlsanity.com/paper/2306.01015/) (2023)
- [A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model](https://mlsanity.com/paper/2305.11244/) (2023)
- [How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario](https://mlsanity.com/paper/2411.18217/) (2024)
- [Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters](https://mlsanity.com/paper/2401.05111/) (2024)
- [PEFT for Speech: Unveiling Optimal Placement, Merging Strategies, and Ensemble Techniques](https://mlsanity.com/paper/2401.02122/) (2024)
- [Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification](https://mlsanity.com/paper/2403.00293/) (2024)
- [JOOCI: a Framework for Learning Comprehensive Speech Representations](https://mlsanity.com/paper/2410.11086/) (2024)
- [UniPET-SPK: A Unified Framework for Parameter-Efficient Tuning of Pre-trained Speech Models for Robust Speaker Verification](https://mlsanity.com/paper/2501.16542/) (2025)
- [COLORA: Efficient Fine-Tuning for Convolutional Models with a Study Case on Optical Coherence Tomography Image Classification](https://mlsanity.com/paper/2505.18315/) (2025)
- [On-the-fly Routing for Zero-shot MoE Speaker Adaptation of Speech Foundation Models for Dysarthric Speech Recognition](https://mlsanity.com/paper/2505.22072/) (2025)
- [G-IFT: A Gated Linear Unit adapter with Iterative Fine-Tuning for Low-Resource Children's Speaker Verification](https://mlsanity.com/paper/2508.07836/) (2025)

## Abstract (arXiv)

Self-supervised learning (SSL) is a powerful technique for learning representations from unlabeled data. Transformer based models such as HuBERT, which consist a feature extractor and transformer layers, are leading the field in the speech domain. SSL models are fine-tuned on a wide range of downstream tasks, which involves re-training the majority of the model for each task. Previous studies have introduced applying adapters, which are small lightweight modules commonly used in Natural Language Processing (NLP) to adapt pre-trained models to new tasks. However, such efficient tuning techniques only provide adaptation at the transformer layer, but failed to perform adaptation at the feature extractor. In this paper, we propose CHAPTER, an efficient tuning method specifically designed for SSL speech model, by applying CNN adapters at the feature extractor. Using this method, we can only fine-tune fewer than 5% of parameters per task compared to fully fine-tuning and achieve better and more stable performance. We empirically found that adding CNN adapters to the feature extractor can help the adaptation on emotion and speaker tasks. For instance, the accuracy of SID is improved from 87.71 to 91.56, and the accuracy of ER is improved by 5%.