mlsanityData to 14 Oct 2026

Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation

arXiv 2509.17349 ↗First posted 22 Sep 2025cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Simultaneous speech-to-text translation systems must balance translation quality with latency. Although quality evaluation is well established, latency measurement remains a challenge. Existing metrics produce inconsistent results, especially in short-form settings with artificial presegmentation. We present the first comprehensive meta-evaluation of latency metrics across language pairs and systems. We uncover a structural bias in current metrics related to segmentation. We introduce YAAL (Yet Another Average Lagging) for a more accurate short-form evaluation and LongYAAL for unsegmented audio. We propose SoftSegmenter, a resegmentation tool based on soft word-level alignment. We show that YAAL and LongYAAL, together with SoftSegmenter, outperform popular latency metrics, enabling more reliable assessments of short- and long-form simultaneous speech translation systems. We implement all artifacts within the OmniSTEval toolkit: https://github.com/pe-trik/OmniSTEval.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

19 of 31 references.

References · 19

  1. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  2. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  3. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  4. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  5. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  6. 2016Can neural machine translation do simultaneous translation?cited by 86
  7. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 57
  8. 2022CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022cited by 30
  9. 2019Thinking Slow about Latency Evaluation for Simultaneous Machine Translationcited by 28
  10. 2024StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selectioncited by 19
  11. 2024Findings of the IWSLT 2024 Evaluation Campaigncited by 19
  12. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
7 more
  1. 2022Average Token Delay: A Latency Metric for Simultaneous Translationcited by 15
  2. 2021Stream-level Latency Evaluation for Simultaneous Machine Translationcited by 12
  3. 2023End-to-End Evaluation for Low-Latency Simultaneous Speech Translationcited by 10
  4. 2024Pitfalls and Outlooks in Using COMETcited by 10
  5. 2022Don't Discard Fixed-Window Audio Segmentation in Speech-to-Text Translationcited by 8
  6. 2023Long-Form End-to-End Speech Translation via Latent Alignment Segmentationcited by 6
  7. 2024Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translationcited by 1
  2. 2026A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026cited by 0
  3. 2026AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Taskcited by 0
  4. 2026MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation taskcited by 0