# Tight Integrated End-to-End Training for Cascaded Speech Translation

arXiv: https://arxiv.org/abs/2011.12167 · HTML: https://mlsanity.com/paper/2011.12167/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2020.

## Built on by (most cited first, 13 of 13)

- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [Efficient yet Competitive Speech Translation: FBK@IWSLT2022](https://mlsanity.com/paper/2205.02629/) (2022)
- [Attention-based Multi-hypothesis Fusion for Speech Summarization](https://mlsanity.com/paper/2111.08201/) (2021)
- [Is Encoder-Decoder Redundant for Neural Machine Translation?](https://mlsanity.com/paper/2210.11807/) (2022)
- [Does Joint Training Really Help Cascaded Speech Translation?](https://mlsanity.com/paper/2210.13700/) (2022)
- [Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation](https://mlsanity.com/paper/2211.05967/) (2022)
- [Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translation](https://mlsanity.com/paper/2304.10295/) (2023)
- [TRAVID: An End-to-End Video Translation Framework](https://mlsanity.com/paper/2309.11338/) (2023)
- [ESPnet-ST IWSLT 2021 Offline Speech Translation System](https://mlsanity.com/paper/2107.00636/) (2021)
- [Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer](https://mlsanity.com/paper/2109.12950/) (2021)
- [End-to-end Training and Decoding for Pivot-based Cascaded Translation Model](https://mlsanity.com/paper/2305.02261/) (2023)
- [Coupling Speech Encoders with Downstream Text Models](https://mlsanity.com/paper/2407.17605/) (2024)

## Abstract (arXiv)

A cascaded speech translation model relies on discrete and non-differentiable transcription, which provides a supervision signal from the source side and helps the transformation between source speech and target text. Such modeling suffers from error propagation between ASR and MT models. Direct speech translation is an alternative method to avoid error propagation; however, its performance is often behind the cascade system. To use an intermediate representation and preserve the end-to-end trainability, previous studies have proposed using two-stage models by passing the hidden vectors of the recognizer into the decoder of the MT model and ignoring the MT encoder. This work explores the feasibility of collapsing the entire cascade components into a single end-to-end trainable model by optimizing all parameters of ASR and MT models jointly without ignoring any learned parameters. It is a tightly integrated method that passes renormalized source word posterior distributions as a soft decision instead of one-hot vectors and enables backpropagation. Therefore, it provides both transcriptions and translations and achieves strong consistency between them. Our experiments on four tasks with different data scenarios show that the model outperforms cascade models up to 1.8% in BLEU and 2.0% in TER and is superior compared to direct models.