# Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognition

arXiv: https://arxiv.org/abs/2012.01687 · HTML: https://mlsanity.com/paper/2012.01687/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 30 of 35)

- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.03218/) (2022)
- [MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognition](https://mlsanity.com/paper/2302.13750/) (2023)
- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2109.09161/) (2021)
- [Improving CTC-based speech recognition via knowledge transferring from pre-trained language models](https://mlsanity.com/paper/2203.03582/) (2022)
- [Automatic Speech Recognition Datasets in Cantonese: A Survey and New Dataset](https://mlsanity.com/paper/2201.02419/) (2022)
- [Distilling a Pretrained Language Model to a Multilingual ASR Model](https://mlsanity.com/paper/2206.12638/) (2022)
- [Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation](https://mlsanity.com/paper/2309.03019/) (2023)
- [Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning](https://mlsanity.com/paper/2302.11343/) (2023)
- [Multilingual Transfer Learning for Code-Switched Language and Speech Neural Modeling](https://mlsanity.com/paper/2104.06268/) (2021)
- [Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters](https://mlsanity.com/paper/2212.10448/) (2022)
- [Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR](https://mlsanity.com/paper/2401.08992/) (2024)
- [Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model](https://mlsanity.com/paper/2309.13018/) (2023)
- [Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification](https://mlsanity.com/paper/2403.00293/) (2024)
- [ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks](https://mlsanity.com/paper/2407.21066/) (2024)
- [Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognition](https://mlsanity.com/paper/2303.01249/) (2023)
- [Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation](https://mlsanity.com/paper/2306.08588/) (2023)
- [What Causes Knowledge Loss in Multilingual Language Models?](https://mlsanity.com/paper/2504.20356/) (2025)
- [Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition](https://mlsanity.com/paper/2210.03255/) (2022)
- [Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition](https://mlsanity.com/paper/2301.00066/) (2022)
- [HTEC: Human Transcription Error Correction](https://mlsanity.com/paper/2309.10089/) (2023)
- [An Effective Context-Balanced Adaptation Approach for Long-Tailed Speech Recognition](https://mlsanity.com/paper/2409.06468/) (2024)
- [UniPET-SPK: A Unified Framework for Parameter-Efficient Tuning of Pre-trained Speech Models for Robust Speaker Verification](https://mlsanity.com/paper/2501.16542/) (2025)
- [Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition](https://mlsanity.com/paper/2603.17558/) (2026)
- [Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR](https://mlsanity.com/paper/2609.15758/) (2026)

## Abstract (arXiv)

One crucial challenge of real-world multilingual speech recognition is the long-tailed distribution problem, where some resource-rich languages like English have abundant training data, but a long tail of low-resource languages have varying amounts of limited training data. To overcome the long-tail problem, in this paper, we propose Adapt-and-Adjust (A2), a transformer-based multi-task learning framework for end-to-end multilingual speech recognition. The A2 framework overcomes the long-tail problem via three techniques: (1) exploiting a pretrained multilingual language model (mBERT) to improve the performance of low-resource languages; (2) proposing dual adapters consisting of both language-specific and language-agnostic adaptation with minimal additional parameters; and (3) overcoming the class imbalance, either by imposing class priors in the loss during training or adjusting the logits of the softmax output during inference. Extensive experiments on the CommonVoice corpus show that A2 significantly outperforms conventional approaches.