# MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

arXiv: https://arxiv.org/abs/2211.08633 · HTML: https://mlsanity.com/paper/2211.08633/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [Unbabel's Participation in the WMT20 Metrics Shared Task](https://mlsanity.com/paper/2010.15535/) (2020)
- [CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022](https://mlsanity.com/paper/2204.06028/) (2022)
- [SHAS: Approaching optimal Segmentation for End-to-End Speech Translation](https://mlsanity.com/paper/2202.04774/) (2022)
- [Efficient yet Competitive Speech Translation: FBK@IWSLT2022](https://mlsanity.com/paper/2205.02629/) (2022)
- [Lost in Interpreting: Speech Translation from Source or Interpreter?](https://mlsanity.com/paper/2106.09343/) (2021)
- [Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translation](https://mlsanity.com/paper/2203.02458/) (2022)
- [Presenting Simultaneous Translation in Limited Space](https://mlsanity.com/paper/2009.09016/) (2020)
- [A Speech Test Set of Practice Business Presentations with Additional Relevant Texts](https://mlsanity.com/paper/1908.00916/) (2019)

## Built on by (most cited first, 9 of 9)

- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent](https://mlsanity.com/paper/2407.21646/) (2024)
- [Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice](https://mlsanity.com/paper/2507.17527/) (2025)
- [Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025](https://mlsanity.com/paper/2506.17077/) (2025)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [Recent Advances in End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2406.00497/) (2024)
- [Automatic Labelling of Speech Translation Errors](https://mlsanity.com/paper/2606.06047/) (2026)
- [Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Design](https://mlsanity.com/paper/2606.16009/) (2026)
- [MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation task](https://mlsanity.com/paper/2606.17255/) (2026)

## Abstract (arXiv)

There have been several meta-evaluation studies on the correlation between human ratings and offline machine translation (MT) evaluation metrics such as BLEU, chrF2, BertScore and COMET. These metrics have been used to evaluate simultaneous speech translation (SST) but their correlations with human ratings of SST, which has been recently collected as Continuous Ratings (CR), are unclear. In this paper, we leverage the evaluations of candidate systems submitted to the English-German SST task at IWSLT 2022 and conduct an extensive correlation analysis of CR and the aforementioned metrics. Our study reveals that the offline metrics are well correlated with CR and can be reliably used for evaluating machine translation in simultaneous mode, with some limitations on the test set size. We conclude that given the current quality levels of SST, these metrics can be used as proxies for CR, alleviating the need for large scale human evaluation. Additionally, we observe that correlations of the metrics with translation as a reference is significantly higher than with simultaneous interpreting, and thus we recommend the former for reliable evaluation.