# Overcoming Latency Bottlenecks in On-Device Speech Translation: A Cascaded Approach with Alignment-Based Streaming MT

arXiv: https://arxiv.org/abs/2508.13358 · HTML: https://mlsanity.com/paper/2508.13358/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Online and Linear-Time Attention by Enforcing Monotonic Alignments](https://mlsanity.com/paper/1704.00784/) (2017)
- [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017)
- [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018)
- [A Comparative Study on End-to-end Speech to Text Translation](https://mlsanity.com/paper/1911.08870/) (2019)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translation](https://mlsanity.com/paper/1912.03393/) (2019)
- [StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning](https://mlsanity.com/paper/2406.03049/) (2024)
- [Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs](https://mlsanity.com/paper/1509.08881/) (2015)
- [Overcoming the Curse of Sentence Length for Neural Machine Translation using Automatic Segmentation](https://mlsanity.com/paper/1409.1257/) (2014)
- [Sentence Boundary Augmentation For Neural Machine Translation Robustness](https://mlsanity.com/paper/2010.11132/) (2020)
- [Glancing Future for Simultaneous Machine Translation](https://mlsanity.com/paper/2309.06179/) (2023)
- [Replacing Human Audio with Synthetic Audio for On-device Unspoken Punctuation Prediction](https://mlsanity.com/paper/2010.10203/) (2020)
- [Efficient Monotonic Multihead Attention](https://mlsanity.com/paper/2312.04515/) (2023)
- [You May Not Need Attention](https://mlsanity.com/paper/1810.13409/) (2018)
- [Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation](https://mlsanity.com/paper/2203.15479/) (2022)
- [AGADIR: Towards Array-Geometry Agnostic Directional Speech Recognition](https://mlsanity.com/paper/2401.10411/) (2024)
- [Segmentation-Free Streaming Machine Translation](https://mlsanity.com/paper/2309.14823/) (2023)
- [Navigating the Minefield of MT Beam Search in Cascaded Streaming Speech Translation](https://mlsanity.com/paper/2407.11010/) (2024)
- [Transcribing and Translating, Fast and Slow: Joint Speech Translation and Recognition](https://mlsanity.com/paper/2412.15415/) (2024)

## Abstract (arXiv)

This paper tackles several challenges that arise when integrating Automatic Speech Recognition (ASR) and Machine Translation (MT) for real-time, on-device streaming speech translation. Although state-of-the-art ASR systems based on Recurrent Neural Network Transducers (RNN-T) can perform real-time transcription, achieving streaming translation in real-time remains a significant challenge. To address this issue, we propose a simultaneous translation approach that effectively balances translation quality and latency. We also investigate efficient integration of ASR and MT, leveraging linguistic cues generated by the ASR system to manage context and utilizing efficient beam-search pruning techniques such as time-out and forced finalization to maintain system's real-time factor. We apply our approach to an on-device bilingual conversational speech translation and demonstrate that our techniques outperform baselines in terms of latency and quality. Notably, our technique narrows the quality gap with non-streaming translation systems, paving the way for more accurate and efficient real-time speech translation.