# A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

arXiv: https://arxiv.org/abs/2606.15059 · HTML: https://mlsanity.com/paper/2606.15059/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 13 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection](https://mlsanity.com/paper/2406.06097/) (2024)
- [Exploiting Sentence Order in Document Alignment](https://mlsanity.com/paper/2004.14523/) (2020)
- [Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS](https://mlsanity.com/paper/2011.04845/) (2020)
- [Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training](https://mlsanity.com/paper/2010.10048/) (2020)
- [Simultaneous Speech-to-Speech Translation Without Aligned Data](https://mlsanity.com/paper/2602.11072/) (2026)
- [Extending Automatic Machine Translation Evaluation to Book-Length Documents](https://mlsanity.com/paper/2509.17249/) (2025)
- [Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation](https://mlsanity.com/paper/2509.17349/) (2025)

## Built on by (most cited first, 1 of 1)

- [EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpreting](https://mlsanity.com/paper/2609.08171/) (2026)

## Abstract (arXiv)

Simultaneous speech-to-speech translation (SimulS2ST) enables real-time cross-lingual communication, but existing evaluation has focused largely on short or pre-segmented speech rather than long-form, continuous input. Prior approaches are difficult to reproduce and make assumptions that do not hold for end-to-end systems. We present a practical evaluation method for long-form SimulS2ST. Given source speech, pre-segmented source transcripts, and reference translations, we run automatic speech recognition (ASR) and forced alignment on the generated target speech to recover token-level timestamps, then apply a sentence-embedding-based aligner to match the target text to its corresponding source sentences. This enables sentence-level computation of latency and quality metrics, including YAAL and xCOMET, which are then aggregated into final system-level scores. Experiments on representative SimulS2ST systems show that the method is effective in practice and reveal that current systems suffer from substantial latency accumulation on long speech. Code can be found here https://github.com/SakaiXue6666/Speech-to-Speech-Latency