# On the Evaluation of Machine Translation for Terminology Consistency

arXiv: https://arxiv.org/abs/2106.11891 · HTML: https://mlsanity.com/paper/2106.11891/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 12 of 12)

- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016)
- [Training Neural Machine Translation To Apply Terminology Constraints](https://mlsanity.com/paper/1906.01105/) (2019)
- [TICO-19: the Translation Initiative for Covid-19](https://mlsanity.com/paper/2007.01788/) (2020)
- [Incorporating Discrete Translation Lexicons into Neural Machine Translation](https://mlsanity.com/paper/1606.02006/) (2016)
- [Facilitating Terminology Translation with Target Lemma Annotations](https://mlsanity.com/paper/2101.10035/) (2021)
- [A Multilingual Neural Machine Translation Model for Biomedical Data](https://mlsanity.com/paper/2008.02878/) (2020)

## Built on by (most cited first, 15 of 15)

- [Benchmarking Machine Translation with Cultural Awareness](https://mlsanity.com/paper/2305.14328/) (2023)
- [BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation](https://mlsanity.com/paper/2305.18326/) (2023)
- [Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus](https://mlsanity.com/paper/2305.11142/) (2023)
- [A Bilingual Parallel Corpus with Discourse Annotations](https://mlsanity.com/paper/2210.14667/) (2022)
- [Dynamic Terminology Integration for COVID-19 and other Emerging Domains](https://mlsanity.com/paper/2109.04708/) (2021)
- [CUNI systems for WMT21: Terminology translation Shared Task](https://mlsanity.com/paper/2109.09350/) (2021)
- [Lingua Custodia's participation at the WMT 2021 Machine Translation using Terminologies shared task](https://mlsanity.com/paper/2111.02120/) (2021)
- [KpopMT: Translation Dataset with Terminology for Kpop Fandom](https://mlsanity.com/paper/2407.07413/) (2024)
- [Disambiguated Lexically Constrained Neural Machine Translation](https://mlsanity.com/paper/2305.17351/) (2023)
- [On Translating Technical Terminology: A Translation Workflow for Machine-Translated Acronyms](https://mlsanity.com/paper/2409.17943/) (2024)
- [Improving Term Evaluation in Machine Translation: Variation Matters](https://mlsanity.com/paper/2609.08779/) (2026)
- [A Template-based Method for Constrained Neural Machine Translation](https://mlsanity.com/paper/2205.11255/) (2022)
- [Cascaded Beam Search: Plug-and-Play Terminology-Forcing For Neural Machine Translation](https://mlsanity.com/paper/2305.14538/) (2023)
- [Improving Neural Machine Translation by Multi-Knowledge Integration with Prompting](https://mlsanity.com/paper/2312.04807/) (2023)
- [TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation](https://mlsanity.com/paper/2609.35017/) (2026)

## Abstract (arXiv)

As neural machine translation (NMT) systems become an important part of professional translator pipelines, a growing body of work focuses on combining NMT with terminologies. In many scenarios and particularly in cases of domain adaptation, one expects the MT output to adhere to the constraints provided by a terminology. In this work, we propose metrics to measure the consistency of MT output with regards to a domain terminology. We perform studies on the COVID-19 domain over 5 languages, also performing terminology-targeted human evaluation. We open-source the code for computing all proposed metrics: https://github.com/mahfuzibnalam/terminology_evaluation