mlsanityData to 14 Oct 2026

From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition

arXiv 2301.07851 ↗First posted 19 Jan 2023cs.SD cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition, which can re-purpose well-trained English automatic speech recognition (ASR) models to recognize the other languages. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement that, for the first time, empowers model reprogramming on ASR. Specifically, we investigate how to select trainable components (i.e., encoder) of a conformer-based RNN-Transducer, as a frozen pre-trained backbone. Experiments on a seven-language multilingual LibriSpeech speech (MLS) task show that model reprogramming only requires 4.2% (11M out of 270M) to 6.8% (45M out of 660M) of its original trainable parameters from a full ASR model to perform competitive results in a range of 11.9% to 8.1% WER averaged across different languages. In addition, we discover different setups to make large-scale pre-trained ASR succeed in both monolingual and multilingual speech recognition. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses (e.g., w2v-bert) in terms of lower WER and better training efficiency.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 19 Jan 2023.

Builds on

31 of 37 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2017Learning multiple visual domains with residual adapterscited by 557
  7. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  11. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  12. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
19 more
  1. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  2. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  3. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  4. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  5. 2018Adversarial Reprogramming of Neural Networkscited by 116
  6. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 93
  7. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  8. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  9. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  10. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  11. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  12. 2020Towards Fast and Accurate Streaming End-to-End ASRcited by 47
  13. 2016Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptationcited by 40
  14. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  15. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 31
  16. 2021On Addressing Practical Challenges for RNN-Transducercited by 19
  17. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  18. 2021Memory-efficient Speech Recognition on Smart Devicescited by 8
  19. 2019Phoneme-Based Contextualization for Cross-Lingual Speech Recognition in End-to-End Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 6; 2025: 3; 2026: 3.

23 papers.

Sort:

By area:

  1. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  2. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 42
  3. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 31
  4. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  5. 2024Sample-specific Masks for Visual Reprogramming-based Promptingcited by 18
  6. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
  7. 2024Bayesian-guided Label Mapping for Visual Reprogrammingcited by 12
  8. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  9. 2023A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Modelcited by 8
  10. 2025Attribute-based Visual Reprogramming for Vision-Language Modelscited by 8
  11. 2023Advancing Test-Time Adaptation in Wild Acoustic Test Settingscited by 7
  12. 2023Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Trainingcited by 4
11 more
  1. 2023Parameter-Efficient Learning for Text-to-Speech Accent Adaptationcited by 4
  2. 2024Exploration of Adapter for Noise Robust Automatic Speech Recognitioncited by 4
  3. 2025Understanding Model Reprogramming for CLIP via Decoupling Visual Promptscited by 4
  4. 2023Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Modelcited by 3
  5. 2025Model Reprogramming Demystified: A Neural Tangent Kernel Perspectivecited by 2
  6. 2023Differentially Private Adapters for Parameter Efficient Acoustic Modelingcited by 1
  7. 2024Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptationcited by 1
  8. 2024An Efficient Self-Learning Framework For Interactive Spoken Dialog Systemscited by 0
  9. 2026Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languagescited by 0
  10. 2026BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogrammingcited by 0
  11. 2026Reprogramming Vision-Language Models via Structured Prompt Reparameterizationcited by 0