# From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition

arXiv: https://arxiv.org/abs/2301.07851 · HTML: https://mlsanity.com/paper/2301.07851/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 19 Jan 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning](https://mlsanity.com/paper/1611.05594/) (2016)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [Pre-training on high-resource speech recognition improves low-resource speech-to-text translation](https://mlsanity.com/paper/1809.01431/) (2018)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [Joint Unsupervised and Supervised Training for Multilingual ASR](https://mlsanity.com/paper/2111.08137/) (2021)
- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks](https://mlsanity.com/paper/2203.16773/) (2022)
- [Memory-efficient Speech Recognition on Smart Devices](https://mlsanity.com/paper/2102.11531/) (2021)

## Built on by (most cited first, 23 of 23)

- [Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition](https://mlsanity.com/paper/2310.06434/) (2023)
- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [Large Language Models are Efficient Learners of Noise-Robust Speech Recognition](https://mlsanity.com/paper/2401.10446/) (2024)
- [SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts](https://mlsanity.com/paper/2306.02207/) (2023)
- [Sample-specific Masks for Visual Reprogramming-based Prompting](https://mlsanity.com/paper/2406.03150/) (2024)
- [Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition](https://mlsanity.com/paper/2309.15223/) (2023)
- [Bayesian-guided Label Mapping for Visual Reprogramming](https://mlsanity.com/paper/2410.24018/) (2024)
- [How to Estimate Model Transferability of Pre-Trained Speech Models?](https://mlsanity.com/paper/2306.01015/) (2023)
- [A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model](https://mlsanity.com/paper/2305.11244/) (2023)
- [Attribute-based Visual Reprogramming for Vision-Language Models](https://mlsanity.com/paper/2501.13982/) (2025)
- [Advancing Test-Time Adaptation in Wild Acoustic Test Settings](https://mlsanity.com/paper/2310.09505/) (2023)
- [Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training](https://mlsanity.com/paper/2303.00786/) (2023)
- [Parameter-Efficient Learning for Text-to-Speech Accent Adaptation](https://mlsanity.com/paper/2305.11320/) (2023)
- [Exploration of Adapter for Noise Robust Automatic Speech Recognition](https://mlsanity.com/paper/2402.18275/) (2024)
- [Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts](https://mlsanity.com/paper/2506.01000/) (2025)
- [Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Model](https://mlsanity.com/paper/2310.02971/) (2023)
- [Model Reprogramming Demystified: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2506.00620/) (2025)
- [Differentially Private Adapters for Parameter Efficient Acoustic Modeling](https://mlsanity.com/paper/2305.11360/) (2023)
- [Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptation](https://mlsanity.com/paper/2406.17257/) (2024)
- [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024)
- [Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languages](https://mlsanity.com/paper/2606.09535/) (2026)
- [BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming](https://mlsanity.com/paper/2606.24740/) (2026)
- [Reprogramming Vision-Language Models via Structured Prompt Reparameterization](https://mlsanity.com/paper/2609.36680/) (2026)

## Abstract (arXiv)

In this work, we propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition, which can re-purpose well-trained English automatic speech recognition (ASR) models to recognize the other languages. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement that, for the first time, empowers model reprogramming on ASR. Specifically, we investigate how to select trainable components (i.e., encoder) of a conformer-based RNN-Transducer, as a frozen pre-trained backbone. Experiments on a seven-language multilingual LibriSpeech speech (MLS) task show that model reprogramming only requires 4.2% (11M out of 270M) to 6.8% (45M out of 660M) of its original trainable parameters from a full ASR model to perform competitive results in a range of 11.9% to 8.1% WER averaged across different languages. In addition, we discover different setups to make large-scale pre-trained ASR succeed in both monolingual and multilingual speech recognition. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses (e.g., w2v-bert) in terms of lower WER and better training efficiency.