# It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data

arXiv: https://arxiv.org/abs/2110.05213 · HTML: https://mlsanity.com/paper/2110.05213/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 7 of 7)

- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [BSTC: A Large-Scale Chinese-English Speech Translation Dataset](https://mlsanity.com/paper/2104.03575/) (2021)
- [Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS](https://mlsanity.com/paper/2011.04845/) (2020)
- [Learning Coupled Policies for Simultaneous Machine Translation using Imitation Learning](https://mlsanity.com/paper/2002.04306/) (2020)

## Built on by (most cited first, 7 of 7)

- [Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair](https://mlsanity.com/paper/2404.12299/) (2024)
- [NAIST-SIC-Aligned: an Aligned English-Japanese Simultaneous Interpretation Corpus](https://mlsanity.com/paper/2304.11766/) (2023)
- [Fixed and Adaptive Simultaneous Machine Translation Strategies Using Adapters](https://mlsanity.com/paper/2407.13469/) (2024)
- [A Major Obstacle for NLP Research: Let's Talk about Time Allocation!](https://mlsanity.com/paper/2211.16858/) (2022)
- [Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies](https://mlsanity.com/paper/2601.11002/) (2026)
- [Rethinking the Reasonability of the Test Set for Simultaneous Machine Translation](https://mlsanity.com/paper/2303.00969/) (2023)
- [Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies](https://mlsanity.com/paper/2509.21801/) (2025)

## Abstract (arXiv)

Most existing simultaneous machine translation (SiMT) systems are trained and evaluated on offline translation corpora. We argue that SiMT systems should be trained and tested on real interpretation data. To illustrate this argument, we propose an interpretation test set and conduct a realistic evaluation of SiMT trained on offline translations. Our results, on our test set along with 3 existing smaller scale language pairs, highlight the difference of up-to 13.83 BLEU score when SiMT models are evaluated on translation vs interpretation data. In the absence of interpretation training data, we propose a translation-to-interpretation (T2I) style transfer method which allows converting existing offline translations into interpretation-style data, leading to up-to 2.8 BLEU improvement. However, the evaluation gap remains notable, calling for constructing large-scale interpretation corpora better suited for evaluating and developing SiMT systems.