# One-To-Many Multilingual End-to-end Speech Translation

arXiv: https://arxiv.org/abs/1910.03320 · HTML: https://mlsanity.com/paper/1910.03320/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 17 of 17)

- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://mlsanity.com/paper/2104.06678/) (2021)
- [AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation](https://mlsanity.com/paper/2312.02512/) (2023)
- [Mukayese: Turkish NLP Strikes Back](https://mlsanity.com/paper/2203.01215/) (2022)
- [End-to-End Speech-to-Text Translation: A Survey](https://mlsanity.com/paper/2312.01053/) (2023)
- [Distilling a Pretrained Language Model to a Multilingual ASR Model](https://mlsanity.com/paper/2206.12638/) (2022)
- [Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation](https://mlsanity.com/paper/2210.09556/) (2022)
- [Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation](https://mlsanity.com/paper/2502.17380/) (2025)
- [How To Build Competitive Multi-gender Speech Translation Models For Controlling Speaker Gender Translation](https://mlsanity.com/paper/2310.15114/) (2023)
- [Efficient Speech Translation with Pre-trained Models](https://mlsanity.com/paper/2211.04939/) (2022)
- [Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation](https://mlsanity.com/paper/2302.10871/) (2023)
- [SBAAM! Eliminating Transcript Dependency in Automatic Subtitling](https://mlsanity.com/paper/2405.10741/) (2024)
- [Selective Data Augmentation for Robust Speech Translation](https://mlsanity.com/paper/2304.03169/) (2023)
- [Joint Training And Decoding for Multilingual End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2503.11080/) (2025)
- [AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2503.14185/) (2025)

## Abstract (arXiv)

Nowadays, training end-to-end neural models for spoken language translation (SLT) still has to confront with extreme data scarcity conditions. The existing SLT parallel corpora are indeed orders of magnitude smaller than those available for the closely related tasks of automatic speech recognition (ASR) and machine translation (MT), which usually comprise tens of millions of instances. To cope with data paucity, in this paper we explore the effectiveness of transfer learning in end-to-end SLT by presenting a multilingual approach to the task. Multilingual solutions are widely studied in MT and usually rely on ``textittarget forcing'', in which multilingual parallel data are combined to train a single model by prepending to the input sequences a language token that specifies the target language. However, when tested in speech translation, our experiments show that MT-like textittarget forcing, used as is, is not effective in discriminating among the target languages. Thus, we propose a variant that uses target-language embeddings to shift the input representations in different portions of the space according to the language, so to better support the production of output in the desired target language. Our experiments on end-to-end SLT from English into six languages show important improvements when translating into similar languages, especially when these are supported by scarce data. Further improvements are obtained when using English ASR data as an additional language (up to +2.5 BLEU points).