# Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS

arXiv: https://arxiv.org/abs/2011.04845 · HTML: https://mlsanity.com/paper/2011.04845/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 12 of 12)

- [A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2406.06937/) (2024)
- [Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention](https://mlsanity.com/paper/2110.08250/) (2021)
- [Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input](https://mlsanity.com/paper/2102.09914/) (2021)
- [SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory](https://mlsanity.com/paper/2108.13630/) (2021)
- [It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data](https://mlsanity.com/paper/2110.05213/) (2021)
- [Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems](https://mlsanity.com/paper/2512.17648/) (2025)
- [Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network](https://mlsanity.com/paper/2109.10724/) (2021)
- [Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representations](https://mlsanity.com/paper/2210.15131/) (2022)
- [QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2309.00126/) (2023)
- [SimulTron: On-Device Simultaneous Speech to Speech Translation](https://mlsanity.com/paper/2406.02133/) (2024)
- [A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2606.15059/) (2026)
- [End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation](https://mlsanity.com/paper/2603.01382/) (2026)

## Abstract (arXiv)

This paper presents a newly developed, simultaneous neural speech-to-speech translation system and its evaluation. The system consists of three fully-incremental neural processing modules for automatic speech recognition (ASR), machine translation (MT), and text-to-speech synthesis (TTS). We investigated its overall latency in the system's Ear-Voice Span and speaking latency along with module-level performance.