# Multilingual Speech Translation with Efficient Finetuning of Pretrained Models

arXiv: https://arxiv.org/abs/2010.12829 · HTML: https://mlsanity.com/paper/2010.12829/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 4 of 4)

- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition](https://mlsanity.com/paper/2101.06699/) (2021)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages](https://mlsanity.com/paper/2411.07387/) (2024)

## Abstract (arXiv)

We present a simple yet effective approach to build multilingual speech-to-text (ST) translation by efficient transfer learning from pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability by only finetuning less than 10% of the pretrained parameters. This enables effectively leveraging large pretrained models with low training cost. Using wav2vec 2.0 for acoustic modeling, and mBART for multilingual text generation, our approach advanced the new state-of-the-art for 34 translation directions (and surpassing cascaded ST for 23 of them) on large-scale multilingual ST benchmark CoVoST 2 (+6.4 BLEU on average across 15 En-X directions and +5.1 BLEU on average across 19 X-En directions). Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model (+5.7 BLEU on average across 18 non-English directions), making it an appealing approach for attaining high-quality speech translation with improved parameter and data efficiency.