# Cascaded Multi-task Adaptive Learning Based on Neural Architecture Search

arXiv: https://arxiv.org/abs/2310.17664 · HTML: https://mlsanity.com/paper/2310.17664/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Incremental Learning Through Deep Adaptation](https://mlsanity.com/paper/1705.04228/) (2017)
- [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022)
- [Universal representations:The missing link between faces, text, planktons, and cat breeds](https://mlsanity.com/paper/1701.07275/) (2017)
- [Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.03218/) (2022)
- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [Towards Better Domain Adaptation for Self-supervised Models: A Case Study of Child ASR](https://mlsanity.com/paper/2305.00115/) (2023)
- [An Adapter Based Pre-Training for Efficient and Scalable Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2107.13530/) (2021)
- [End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representation](https://mlsanity.com/paper/2204.00540/) (2022)
- [Layer-wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition](https://mlsanity.com/paper/2204.09883/) (2022)
- [Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoders](https://mlsanity.com/paper/2011.01097/) (2020)
- [What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning](https://mlsanity.com/paper/2007.12415/) (2020)
- [Adaptable Multi-Domain Language Model for Transformer ASR](https://mlsanity.com/paper/2008.06208/) (2020)

## Abstract (arXiv)

Cascading multiple pre-trained models is an effective way to compose an end-to-end system. However, fine-tuning the full cascaded model is parameter and memory inefficient and our observations reveal that only applying adapter modules on cascaded model can not achieve considerable performance as fine-tuning. We propose an automatic and effective adaptive learning method to optimize end-to-end cascaded multi-task models based on Neural Architecture Search (NAS) framework. The candidate adaptive operations on each specific module consist of frozen, inserting an adapter and fine-tuning. We further add a penalty item on the loss to limit the learned structure which takes the amount of trainable parameters into account. The penalty item successfully restrict the searched architecture and the proposed approach is able to search similar tuning scheme with hand-craft, compressing the optimizing parameters to 8.7% corresponding to full fine-tuning on SLURP with an even better performance.