# Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning

arXiv: https://arxiv.org/abs/2306.15686 · HTML: https://mlsanity.com/paper/2306.15686/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 30 of 32)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [Multilingual Speech Recognition With A Single End-To-End Model](https://mlsanity.com/paper/1711.01694/) (2017)
- [Modular Networks: Learning to Decompose Neural Computation](https://mlsanity.com/paper/1811.05249/) (2018)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Speech Recognition Challenge in the Wild: Arabic MGB-3](https://mlsanity.com/paper/1709.07276/) (2017)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [Low-Resource Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2002.10348/) (2020)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Meta Learning for End-to-End Low-Resource Speech Recognition](https://mlsanity.com/paper/1910.12094/) (2019)
- [Combining Modular Skills in Multitask Learning](https://mlsanity.com/paper/2202.13914/) (2022)
- [Training Your Sparse Neural Network Better with Any Mask](https://mlsanity.com/paper/2206.12755/) (2022)
- [Meta-Transfer Learning for Code-Switched Speech Recognition](https://mlsanity.com/paper/2004.14228/) (2020)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2109.09161/) (2021)
- [Towards Zero-shot Learning for Automatic Phonemic Transcription](https://mlsanity.com/paper/2002.11781/) (2020)
- [Multilingual and crosslingual speech recognition using phonological-vector based phone embeddings](https://mlsanity.com/paper/2107.05038/) (2021)
- [Adaptive multilingual speech recognition with pretrained models](https://mlsanity.com/paper/2205.12304/) (2022)
- [Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance](https://mlsanity.com/paper/2203.02770/) (2022)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)

## Built on by (most cited first, 9 of 9)

- [Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration](https://mlsanity.com/paper/2406.15765/) (2024)
- [EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting](https://mlsanity.com/paper/2406.15758/) (2024)
- [Wav2Gloss: Generating Interlinear Glossed Text from Speech](https://mlsanity.com/paper/2403.13169/) (2024)
- [Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems](https://mlsanity.com/paper/2407.03645/) (2024)
- [Exploration of Adapter for Noise Robust Automatic Speech Recognition](https://mlsanity.com/paper/2402.18275/) (2024)
- [A Parameter-efficient Language Extension Framework for Multilingual ASR](https://mlsanity.com/paper/2406.06329/) (2024)
- [Automatic Speech Recognition Advancements for Indigenous Languages of the Americas](https://mlsanity.com/paper/2404.08368/) (2024)
- [Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR](https://mlsanity.com/paper/2410.13445/) (2024)
- [Task Arithmetic with Support Languages for Low-Resource ASR](https://mlsanity.com/paper/2601.07038/) (2026)

## Abstract (arXiv)

Despite the impressive performance recently achieved by automatic speech recognition (ASR), we observe two primary challenges that hinder its broader applications: (1) The difficulty of introducing scalability into the model to support more languages with limited training, inference, and storage overhead; (2) The low-resource adaptation ability that enables effective low-resource adaptation while avoiding over-fitting and catastrophic forgetting issues. Inspired by recent findings, we hypothesize that we can address the above challenges with modules widely shared across languages. To this end, we propose an ASR framework, dubbed METHODNS, that, textitfor the first time, simultaneously achieves strong multilingual scalability and low-resource adaptation ability thanks to its modularize-then-assemble strategy. Specifically, METHOD learns a small set of generalizable sub-modules and adaptively assembles them for different languages to reduce the multilingual overhead and enable effective knowledge transfer for low-resource adaptation. Extensive experiments and visualizations demonstrate that METHOD can effectively discover language similarity and improve multilingual and low-resource ASR performance over state-of-the-art (SOTA) methods, e.g., under multilingual-ASR, our framework achieves a 0.13~2.41 lower character error rate (CER) with 30% smaller inference overhead over SOTA solutions on multilingual ASR and a comparable CER, with nearly 50 times fewer trainable parameters over SOTA solutions on low-resource tuning, respectively.