# Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models

arXiv: https://arxiv.org/abs/2403.19709 · HTML: https://mlsanity.com/paper/2403.19709/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [Light Gated Recurrent Units for Speech Recognition](https://mlsanity.com/paper/1803.10225/) (2018)
- [Toward domain-invariant speech recognition via large scale training](https://mlsanity.com/paper/1808.05312/) (2018)
- [Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models](https://mlsanity.com/paper/2203.03131/) (2022)
- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [Pseudo Label Is Better Than Human Label](https://mlsanity.com/paper/2203.12668/) (2022)
- [Deep Independently Recurrent Neural Network (IndRNN)](https://mlsanity.com/paper/1910.06251/) (2019)
- [READ: Recurrent Adaptation of Large Transformers](https://mlsanity.com/paper/2305.15348/) (2023)
- [A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization](https://mlsanity.com/paper/2203.12559/) (2022)
- [Modular Domain Adaptation for Conformer-Based Streaming ASR](https://mlsanity.com/paper/2305.13408/) (2023)
- [Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling](https://mlsanity.com/paper/2009.01803/) (2020)
- [Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion](https://mlsanity.com/paper/2211.02712/) (2022)

## Built on by (most cited first, 2 of 2)

- [Towards Optimal Adapter Placement for Efficient Transfer Learning](https://mlsanity.com/paper/2410.15858/) (2024)
- [SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning](https://mlsanity.com/paper/2608.15360/) (2026)

## Abstract (arXiv)

Parameter efficient adaptation methods have become a key mechanism to train large pre-trained models for downstream tasks. However, their per-task parameter overhead is considered still high when the number of downstream tasks to adapt for is large. We introduce an adapter module that has a better efficiency in large scale multi-task adaptation scenario. Our adapter is hierarchical in terms of how the adapter parameters are allocated. The adapter consists of a single shared controller network and multiple task-level adapter heads to reduce the per-task parameter overhead without performance regression on downstream tasks. The adapter is also recurrent so the entire adapter parameters are reused across different layers of the pre-trained model. Our Hierarchical Recurrent Adapter (HRA) outperforms the previous adapter-based approaches as well as full model fine-tuning baseline in both single and multi-task adaptation settings when evaluated on automatic speech recognition tasks.