# Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation

arXiv: https://arxiv.org/abs/2509.17349 · HTML: https://mlsanity.com/paper/2509.17349/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation](https://mlsanity.com/paper/2206.05807/) (2022)
- [CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022](https://mlsanity.com/paper/2204.06028/) (2022)
- [Thinking Slow about Latency Evaluation for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.00048/) (2019)
- [StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection](https://mlsanity.com/paper/2406.06097/) (2024)
- [Findings of the IWSLT 2024 Evaluation Campaign](https://mlsanity.com/paper/2411.05088/) (2024)
- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Average Token Delay: A Latency Metric for Simultaneous Translation](https://mlsanity.com/paper/2211.13173/) (2022)
- [Stream-level Latency Evaluation for Simultaneous Machine Translation](https://mlsanity.com/paper/2104.08817/) (2021)
- [End-to-End Evaluation for Low-Latency Simultaneous Speech Translation](https://mlsanity.com/paper/2308.03415/) (2023)
- [Pitfalls and Outlooks in Using COMET](https://mlsanity.com/paper/2408.15366/) (2024)
- [Don't Discard Fixed-Window Audio Segmentation in Speech-to-Text Translation](https://mlsanity.com/paper/2210.13363/) (2022)
- [Long-Form End-to-End Speech Translation via Latent Alignment Segmentation](https://mlsanity.com/paper/2309.11384/) (2023)
- [Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation](https://mlsanity.com/paper/2406.03881/) (2024)

## Built on by (most cited first, 4 of 4)

- [A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2606.15059/) (2026)
- [A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026](https://mlsanity.com/paper/2606.03948/) (2026)
- [AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task](https://mlsanity.com/paper/2606.03967/) (2026)
- [MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation task](https://mlsanity.com/paper/2606.17255/) (2026)

## Abstract (arXiv)

Simultaneous speech-to-text translation systems must balance translation quality with latency. Although quality evaluation is well established, latency measurement remains a challenge. Existing metrics produce inconsistent results, especially in short-form settings with artificial presegmentation. We present the first comprehensive meta-evaluation of latency metrics across language pairs and systems. We uncover a structural bias in current metrics related to segmentation. We introduce YAAL (Yet Another Average Lagging) for a more accurate short-form evaluation and LongYAAL for unsegmented audio. We propose SoftSegmenter, a resegmentation tool based on soft word-level alignment. We show that YAAL and LongYAAL, together with SoftSegmenter, outperform popular latency metrics, enabling more reliable assessments of short- and long-form simultaneous speech translation systems. We implement all artifacts within the OmniSTEval toolkit: https://github.com/pe-trik/OmniSTEval.