# EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpreting

arXiv: https://arxiv.org/abs/2609.08171 · HTML: https://mlsanity.com/paper/2609.08171/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning](https://mlsanity.com/paper/2406.03049/) (2024)
- [Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation](https://mlsanity.com/paper/2401.06568/) (2024)
- [Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent](https://mlsanity.com/paper/2407.21646/) (2024)
- [M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation](https://mlsanity.com/paper/2412.20127/) (2024)
- [The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control](https://mlsanity.com/paper/2405.16969/) (2024)
- [Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translation](https://mlsanity.com/paper/2203.02458/) (2022)
- [Exploring the Correlation between Human and Machine Evaluation of Simultaneous Speech Translation](https://mlsanity.com/paper/2406.10091/) (2024)
- [Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies](https://mlsanity.com/paper/2601.11002/) (2026)
- [A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2606.15059/) (2026)

## Abstract (arXiv)

Low-latency simultaneous speech-to-speech translation must keep pace with ongoing speech while preserving key information. To meet these demands, systems use segmentation, reformulation and condensation to reorganize and rephrase information. However, metrics developed for text translation, including BLEU and COMET, may not consistently distinguish faithful adaptations from semantic errors. We propose EviSI, a large language model evaluation agent combining Multidimensional Quality Metrics (MQM) with criteria developed with professional interpreters. Shared source evidence guides assessment across four dimensions: Anchor, Event, Logic and Fluency. Verified errors are deduplicated before deterministic scoring. On human-rated English to Chinese and Chinese to English data, EviSI recovers the aggregate English to Chinese human system ranking. Mean within-dataset Kendall correlations for system rankings reach 0.707 and 0.467, respectively, exceeding evaluated BLEU and COMET baselines. A multilingual extension to five directions without human ratings retains the dimensions and scoring rule, showing positive system ranking correlations with COMET throughout.