# Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training

arXiv: https://arxiv.org/abs/2010.10048 · HTML: https://mlsanity.com/paper/2010.10048/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 10 of 10)

- [Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation](https://mlsanity.com/paper/2102.05766/) (2021)
- [Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy](https://mlsanity.com/paper/2109.05238/) (2021)
- [Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR](https://mlsanity.com/paper/2106.06636/) (2021)
- [A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2406.06937/) (2024)
- [Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention](https://mlsanity.com/paper/2110.08250/) (2021)
- [Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network](https://mlsanity.com/paper/2109.10724/) (2021)
- [Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approach](https://mlsanity.com/paper/2310.04399/) (2023)
- [A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2606.15059/) (2026)
- [Toward Machine Interpreting: Lessons from Human Interpreting Studies](https://mlsanity.com/paper/2508.07964/) (2025)
- [Simultaneous Translation between Sign Languages](https://mlsanity.com/paper/2609.35608/) (2026)

## Abstract (arXiv)

Simultaneous speech-to-speech translation is widely useful but extremely challenging, since it needs to generate target-language speech concurrently with the source-language speech, with only a few seconds delay. In addition, it needs to continuously translate a stream of sentences, but all recent solutions merely focus on the single-sentence scenario. As a result, current approaches accumulate latencies progressively when the speaker talks faster, and introduce unnatural pauses when the speaker talks slower. To overcome these issues, we propose Self-Adaptive Translation (SAT) which flexibly adjusts the length of translations to accommodate different source speech rates. At similar levels of translation quality (as measured by BLEU), our method generates more fluent target speech (as measured by the naturalness metric MOS) with substantially lower latency than the baseline, in both Zh <-> En directions.