mlsanityData to 14 Oct 2026

Efficient Domain Adaptation for Speech Foundation Models

arXiv 2302.01496 ↗First posted 3 Feb 2023cs.CL cs.LG cs.SDcited by 13

Not on the map. Why?

Abstract (arXiv)

Foundation models (FMs), that are trained on broad data at scale and are adaptable to a wide range of downstream tasks, have brought large interest in the research community. Benefiting from the diverse data sources such as different modalities, languages and application domains, foundation models have demonstrated strong generalization and knowledge transfer capabilities. In this paper, we present a pioneering study towards building an efficient solution for FM-based speech recognition systems. We adopt the recently developed self-supervised BEST-RQ for pretraining, and propose the joint finetuning with both source and unsupervised target domain data using JUST Hydra. The FM encoder adapter and decoder are then finetuned to the target domain with a small amount of supervised in-domain data. On a large-scale YouTube and Voice Search task, our method is shown to be both data and model parameter efficient. It achieves the same quality with only 21.6M supervised in-domain data and 130.8M finetuned parameters, compared to the 731.1M model trained from scratch on additional 300M supervised in-domain data.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

43 of 54 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
31 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  2. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  3. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  4. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  7. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  8. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 101
  9. 2019Deep Learning for Audio Signal Processingcited by 92
  10. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  11. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  12. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  13. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  14. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  15. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  16. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  17. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  18. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
  19. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  20. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  21. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  22. 2018Toward domain-invariant speech recognition via large scale trainingcited by 35
  23. 2020RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutionscited by 26
  24. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  25. 2022Pseudo Label Is Better Than Human Labelcited by 25
  26. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  27. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 23
  28. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  29. 2021Analyzing the Robustness of Unsupervised Speech Recognitioncited by 11
  30. 2022A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalizationcited by 8
  31. 2022Unsupervised data selection for Speech Recognition with contrastive loss ratioscited by 7

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 4; 2025: 1.

13 papers.

Sort:

By area:

  1. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 200
  2. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  3. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  4. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 17
  5. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  6. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  7. 2023Modular Domain Adaptation for Conformer-Based Streaming ASRcited by 7
  8. 2024How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenariocited by 5
  9. 2023Improving Speech Recognition for African American English With Audio Classificationcited by 4
  10. 2024Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASRcited by 4
  11. 2023Differentially Private Adapters for Parameter Efficient Acoustic Modelingcited by 1
  12. 2024Adapter-based Approaches to Knowledge-enhanced Language Models -- A Surveycited by 0
1 more
  1. 2025On-the-fly Routing for Zero-shot MoE Speaker Adaptation of Speech Foundation Models for Dysarthric Speech Recognitioncited by 0