# Visualization: the missing factor in Simultaneous Speech Translation

arXiv: https://arxiv.org/abs/2111.00514 · HTML: https://mlsanity.com/paper/2111.00514/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 31 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020)
- [Streaming automatic speech recognition with the transformer model](https://mlsanity.com/paper/2001.02674/) (2020)
- [Efficient Wait-k Models for Simultaneous Machine Translation](https://mlsanity.com/paper/2005.08595/) (2020)
- [Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://mlsanity.com/paper/2106.01045/) (2021)
- [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018)
- [Simultaneous Translation Policies: From Fixed to Adaptive](https://mlsanity.com/paper/2004.13169/) (2020)
- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [Low-Latency Neural Speech Translation](https://mlsanity.com/paper/1808.00491/) (2018)
- [Simpler and Faster Learning of Adaptive Policies for Simultaneous Translation](https://mlsanity.com/paper/1909.01559/) (2019)
- [Re-translation versus Streaming for Simultaneous Translation](https://mlsanity.com/paper/2004.03643/) (2020)
- [Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR](https://mlsanity.com/paper/2106.06636/) (2021)
- [Thinking Slow about Latency Evaluation for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.00048/) (2019)
- [Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translation](https://mlsanity.com/paper/1912.03393/) (2019)
- [Simultaneous Translation with Flexible Policy via Restricted Imitation Learning](https://mlsanity.com/paper/1906.01135/) (2019)
- [Low Latency End-to-End Streaming Speech Recognition with a Scout Network](https://mlsanity.com/paper/2003.10369/) (2020)
- [An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategies](https://mlsanity.com/paper/2103.03233/) (2021)
- [Speculative Beam Search for Simultaneous Translation](https://mlsanity.com/paper/1909.05421/) (2019)
- [Towards the evaluation of automatic simultaneous speech translation from a communicative perspective](https://mlsanity.com/paper/2103.08364/) (2021)
- [Simultaneous Speech Translation for Live Subtitling: from Delay to Display](https://mlsanity.com/paper/2107.08807/) (2021)
- [DuTongChuan: Context-aware Translation Model for Simultaneous Interpreting](https://mlsanity.com/paper/1907.12984/) (2019)
- [Online Versus Offline NMT Quality: An In-depth Analysis on English-German and German-English](https://mlsanity.com/paper/2006.00814/) (2020)
- [Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2104.14470/) (2021)

## Built on by (most cited first, 2 of 2)

- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)

## Abstract (arXiv)

Simultaneous speech translation (SimulST) is the task in which output generation has to be performed on partial, incremental speech input. In recent years, SimulST has become popular due to the spread of cross-lingual application scenarios, like international live conferences and streaming lectures, in which on-the-fly speech translation can facilitate users' access to audio-visual content. In this paper, we analyze the characteristics of the SimulST systems developed so far, discussing their strengths and weaknesses. We then concentrate on the evaluation framework required to properly assess systems' effectiveness. To this end, we raise the need for a broader performance analysis, also including the user experience standpoint. SimulST systems, indeed, should be evaluated not only in terms of quality/latency measures, but also via task-oriented metrics accounting, for instance, for the visualization strategy adopted. In light of this, we highlight which are the goals achieved by the community and what is still missing.