# What does it take to get state of the art in simultaneous speech-to-speech translation?

arXiv: https://arxiv.org/abs/2409.00965 · HTML: https://mlsanity.com/paper/2409.00965/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020)
- [Efficient Wait-k Models for Simultaneous Machine Translation](https://mlsanity.com/paper/2005.08595/) (2020)
- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [Simultaneous Translation Policies: From Fixed to Adaptive](https://mlsanity.com/paper/2004.13169/) (2020)
- [Simpler and Faster Learning of Adaptive Policies for Simultaneous Translation](https://mlsanity.com/paper/1909.01559/) (2019)
- [Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?](https://mlsanity.com/paper/1905.10617/) (2019)

## Built on by (most cited first, 1 of 1)

- [From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection](https://mlsanity.com/paper/2606.23060/) (2026)

## Abstract (arXiv)

This paper presents an in-depth analysis of the latency characteristics observed in simultaneous speech-to-speech model's performance, particularly focusing on hallucination-induced latency spikes. By systematically experimenting with various input parameters and conditions, we propose methods to minimize latency spikes and improve overall performance. The findings suggest that a combination of careful input management and strategic parameter adjustments can significantly enhance speech-to-speech model's latency behavior.