# Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translation

arXiv: https://arxiv.org/abs/2104.14470 · HTML: https://mlsanity.com/paper/2104.14470/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Efficient Wait-k Models for Simultaneous Machine Translation](https://mlsanity.com/paper/2005.08595/) (2020)
- [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018)
- [Low-Latency Neural Speech Translation](https://mlsanity.com/paper/1808.00491/) (2018)
- [Thinking Slow about Latency Evaluation for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.00048/) (2019)
- [An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategies](https://mlsanity.com/paper/2103.03233/) (2021)
- [ON-TRAC Consortium for End-to-End and Simultaneous Speech Translation Challenge Tasks at IWSLT 2020](https://mlsanity.com/paper/2005.11861/) (2020)
- [Online Versus Offline NMT Quality: An In-depth Analysis on English-German and German-English](https://mlsanity.com/paper/2006.00814/) (2020)
- [ON-TRAC Consortium End-to-End Speech Translation Systems for the IWSLT 2019 Shared Task](https://mlsanity.com/paper/1910.13689/) (2019)

## Built on by (most cited first, 4 of 4)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation](https://mlsanity.com/paper/2204.09595/) (2022)
- [Recent Advances in End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2406.00497/) (2024)
- [Visualization: the missing factor in Simultaneous Speech Translation](https://mlsanity.com/paper/2111.00514/) (2021)

## Abstract (arXiv)

Boosted by the simultaneous translation shared task at IWSLT 2020, promising end-to-end online speech translation approaches were recently proposed. They consist in incrementally encoding a speech input (in a source language) and decoding the corresponding text (in a target language) with the best possible trade-off between latency and translation quality. This paper investigates two key aspects of end-to-end simultaneous speech translation: (a) how to encode efficiently the continuous speech flow, and (b) how to segment the speech flow in order to alternate optimally between reading (R: encoding input) and writing (W: decoding output) operations. We extend our previously proposed end-to-end online decoding strategy and show that while replacing BLSTM by ULSTM encoding degrades performance in offline mode, it actually improves both efficiency and performance in online mode. We also measure the impact of different methods to segment the speech signal (using fixed interval boundaries, oracle word boundaries or randomly set boundaries) and show that our best end-to-end online decoding strategy is surprisingly the one that alternates R/W operations on fixed size blocks on our English-German speech translation setup.