# Does Joint Training Really Help Cascaded Speech Translation?

arXiv: https://arxiv.org/abs/2210.13700 · HTML: https://mlsanity.com/paper/2210.13700/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020)
- [End-to-End Automatic Speech Translation of Audiobooks](https://mlsanity.com/paper/1802.04200/) (2018)
- [Tied Multitask Learning for Neural Speech Translation](https://mlsanity.com/paper/1802.06655/) (2018)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation](https://mlsanity.com/paper/1909.07575/) (2019)
- [Multilingual End-to-End Speech Translation](https://mlsanity.com/paper/1910.00254/) (2019)
- [Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation](https://mlsanity.com/paper/1802.03142/) (2018)
- [Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translation](https://mlsanity.com/paper/1904.07209/) (2019)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [Neural Lattice-to-Sequence Models for Uncertain Inputs](https://mlsanity.com/paper/1704.00559/) (2017)
- [On Using SpecAugment for End-to-End Speech Translation](https://mlsanity.com/paper/1911.08876/) (2019)
- [Lattice Transformer for Speech Translation](https://mlsanity.com/paper/1906.05551/) (2019)
- [Tight Integrated End-to-End Training for Cascaded Speech Translation](https://mlsanity.com/paper/2011.12167/) (2020)
- [Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translation](https://mlsanity.com/paper/1802.06003/) (2018)
- [LibriVoxDeEn: A Corpus for German-to-English Speech Translation and German Speech Recognition](https://mlsanity.com/paper/1910.07924/) (2019)

## Built on by (most cited first, 5 of 5)

- [Towards Speech Dialogue Translation Mediating Speakers of Different Languages](https://mlsanity.com/paper/2305.09210/) (2023)
- [Strategies for improving low resource speech to text translation relying on pre-trained ASR models](https://mlsanity.com/paper/2306.00208/) (2023)
- [MLLP-VRAIN UPV system for the IWSLT 2025 Simultaneous Speech Translation Translation task](https://mlsanity.com/paper/2506.18828/) (2025)
- [BeaverTalk: Oregon State University's IWSLT 2025 Simultaneous Speech Translation System](https://mlsanity.com/paper/2505.24016/) (2025)
- [TokenMapper: A Step Toward Interoperable Speech Token Translation](https://mlsanity.com/paper/2609.12563/) (2026)

## Abstract (arXiv)

Currently, in speech translation, the straightforward approach - cascading a recognition system with a translation system - delivers state-of-the-art results. However, fundamental challenges such as error propagation from the automatic speech recognition system still remain. To mitigate these problems, recently, people turn their attention to direct data and propose various joint training methods. In this work, we seek to answer the question of whether joint training really helps cascaded speech translation. We review recent papers on the topic and also investigate a joint training criterion by marginalizing the transcription posterior probabilities. Our findings show that a strong cascaded baseline can diminish any improvements obtained using joint training, and we suggest alternatives to joint training. We hope this work can serve as a refresher of the current speech translation landscape, and motivate research in finding more efficient and creative ways to utilize the direct data for speech translation.