# The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Task

arXiv: https://arxiv.org/abs/2206.05777 · HTML: https://mlsanity.com/paper/2206.05777/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 12 Jun 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [CoVoST 2 and Massively Multilingual Speech-to-Text Translation](https://mlsanity.com/paper/2007.10310/) (2020)
- [SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training](https://mlsanity.com/paper/2110.10329/) (2021)
- [Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1811.02050/) (2018)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [Shallow-to-Deep Training for Neural Machine Translation](https://mlsanity.com/paper/2010.03737/) (2020)
- [Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task](https://mlsanity.com/paper/2111.02086/) (2021)
- [Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data](https://mlsanity.com/paper/2203.17113/) (2022)
- [FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task](https://mlsanity.com/paper/2107.06959/) (2021)

## Built on by (most cited first, 3 of 3)

- [Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys](https://mlsanity.com/paper/2401.10352/) (2024)
- [Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23](https://mlsanity.com/paper/2306.01327/) (2023)
- [SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation](https://mlsanity.com/paper/2406.14177/) (2024)

## Abstract (arXiv)

This paper describes the submission of our end-to-end YiTrans speech translation system for the IWSLT 2022 offline task, which translates from English audio to German, Chinese, and Japanese. The YiTrans system is built on large-scale pre-trained encoder-decoder models. More specifically, we first design a multi-stage pre-training strategy to build a multi-modality model with a large amount of labeled and unlabeled data. We then fine-tune the corresponding components of the model for the downstream speech translation tasks. Moreover, we make various efforts to improve performance, such as data filtering, data augmentation, speech segmentation, model ensemble, and so on. Experimental results show that our YiTrans system obtains a significant improvement than the strong baseline on three translation directions, and it achieves +5.2 BLEU improvements over last year's optimal end-to-end system on tst2021 English-German. Our final submissions rank first on English-German and English-Chinese end-to-end systems in terms of the automatic evaluation metric. We make our code and models publicly available.