# Long-Form End-to-End Speech Translation via Latent Alignment Segmentation

arXiv: https://arxiv.org/abs/2309.11384 · HTML: https://mlsanity.com/paper/2309.11384/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates](https://mlsanity.com/paper/1911.03167/) (2019)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Speech Translation and the End-to-End Promise: Taking Stock of Where We Are](https://mlsanity.com/paper/2004.06358/) (2020)
- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018)
- [Low-Latency Neural Speech Translation](https://mlsanity.com/paper/1808.00491/) (2018)
- [CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022](https://mlsanity.com/paper/2204.06028/) (2022)
- [SHAS: Approaching optimal Segmentation for End-to-End Speech Translation](https://mlsanity.com/paper/2202.04774/) (2022)
- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation](https://mlsanity.com/paper/2104.11710/) (2021)
- [Stream-level Latency Evaluation for Simultaneous Machine Translation](https://mlsanity.com/paper/2104.08817/) (2021)
- [Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation](https://mlsanity.com/paper/2204.08920/) (2022)
- [ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit](https://mlsanity.com/paper/2304.04596/) (2023)
- [Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation](https://mlsanity.com/paper/2203.15479/) (2022)

## Built on by (most cited first, 6 of 6)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation](https://mlsanity.com/paper/2509.17349/) (2025)
- [Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems](https://mlsanity.com/paper/2512.17648/) (2025)
- [Long-form Simultaneous Speech Translation: Thesis Proposal](https://mlsanity.com/paper/2310.11141/) (2023)
- [Lightweight Audio Segmentation for Long-form Speech Translation](https://mlsanity.com/paper/2406.10549/) (2024)
- [DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs](https://mlsanity.com/paper/2605.31432/) (2026)

## Abstract (arXiv)

Current simultaneous speech translation models can process audio only up to a few seconds long. Contemporary datasets provide an oracle segmentation into sentences based on human-annotated transcripts and translations. However, the segmentation into sentences is not available in the real world. Current speech segmentation approaches either offer poor segmentation quality or have to trade latency for quality. In this paper, we propose a novel segmentation approach for a low-latency end-to-end speech translation. We leverage the existing speech translation encoder-decoder architecture with ST CTC and show that it can perform the segmentation task without supervision or additional parameters. To the best of our knowledge, our method is the first that allows an actual end-to-end simultaneous speech translation, as the same model is used for translation and segmentation at the same time. On a diverse set of language pairs and in- and out-of-domain data, we show that the proposed approach achieves state-of-the-art quality at no additional computational cost.