mlsanityData to 7 Oct 2026

Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR

arXiv 2410.13445 ↗First posted 17 Oct 2024cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Automatic speech recognition (ASR) for low-resource languages remains a challenge due to the scarcity of labeled training data. Parameter-efficient fine-tuning and text-only adaptation are two popular methods that have been used to address such low-resource settings. In this work, we investigate how these techniques can be effectively combined using a multilingual multimodal model like SeamlessM4T. Multimodal models are able to leverage unlabeled text via text-only adaptation with further parameter-efficient ASR fine-tuning, thus boosting ASR performance. We also show cross-lingual transfer from a high-resource language, achieving up to a relative 17% WER reduction over a baseline in a zero-shot setting without any labeled speech.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

28 of 44 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
16 more
  1. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  2. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  3. 2020Low-resource Languages: A Review of Past Work and Future Challengescited by 90
  4. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  5. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  6. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  7. 2017Learning Language Representations for Typology Predictioncited by 40
  8. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  9. 2024IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languagescited by 39
  10. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 33
  11. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 30
  12. 2022A Survey of Multilingual Models for Automatic Speech Recognitioncited by 21
  13. 2019What do Language Representations Really Represent?cited by 17
  14. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  15. 2023Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generatorcited by 11
  16. 2023Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learningcited by 9

Built on by

Not cited yet.