# NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures

arXiv: https://arxiv.org/abs/2204.13692 · HTML: https://mlsanity.com/paper/2204.13692/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://mlsanity.com/paper/1704.04154/) (2017)
- [Explicit Alignment Objectives for Multilingual Bidirectional Encoders](https://mlsanity.com/paper/2010.07972/) (2020)
- [Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora](https://mlsanity.com/paper/1809.00197/) (2018)
- [Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext](https://mlsanity.com/paper/1706.01847/) (2017)
- [Simple and Effective Paraphrastic Similarity from Parallel Translations](https://mlsanity.com/paper/1909.13872/) (2019)
- [SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics](https://mlsanity.com/paper/2007.05374/) (2020)
- [Multilingual AMR-to-Text Generation](https://mlsanity.com/paper/2011.05443/) (2020)
- [Finnish Paraphrase Corpus](https://mlsanity.com/paper/2103.13103/) (2021)
- [Assessing Reference-Free Peer Evaluation for Machine Translation](https://mlsanity.com/paper/2104.05146/) (2021)
- [Are BLEU and Meaning Representation in Opposition?](https://mlsanity.com/paper/1805.06536/) (2018)

## Built on by (most cited first, 6 of 6)

- [T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks](https://mlsanity.com/paper/2212.10548/) (2022)
- [Machine Translation Models are Zero-Shot Detectors of Translation Direction](https://mlsanity.com/paper/2401.06769/) (2024)
- [Multilingual Lexical Simplification via Paraphrase Generation](https://mlsanity.com/paper/2307.15286/) (2023)
- [EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation](https://mlsanity.com/paper/2403.00144/) (2024)
- [Revisiting Projection-based Data Transfer for Cross-Lingual Named Entity Recognition in Low-Resource Languages](https://mlsanity.com/paper/2501.18750/) (2025)
- [Cogs in a Machine, Doing What They're Meant to Do -- The AMI Submission to the WMT24 General Translation Task](https://mlsanity.com/paper/2410.03381/) (2024)

## Abstract (arXiv)

Being able to rank the similarity of short text segments is an interesting bonus feature of neural machine translation. Translation-based similarity measures include direct and pivot translation probability, as well as translation cross-likelihood, which has not been studied so far. We analyze these measures in the common framework of multilingual NMT, releasing the NMTScore library (available at https://github.com/ZurichNLP/nmtscore). Compared to baselines such as sentence embeddings, translation-based measures prove competitive in paraphrase identification and are more robust against adversarial or multilingual input, especially if proper normalization is applied. When used for reference-based evaluation of data-to-text generation in 2 tasks and 17 languages, translation-based measures show a relatively high correlation to human judgments.