# Textual Similarity as a Key Metric in Machine Translation Quality Estimation

arXiv: https://arxiv.org/abs/2406.07440 · HTML: https://mlsanity.com/paper/2406.07440/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://mlsanity.com/paper/2011.01536/) (2020)
- [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://mlsanity.com/paper/2010.04480/) (2020)
- [PreQuEL: Quality Estimation of Machine Translation Outputs in Advance](https://mlsanity.com/paper/2205.09178/) (2022)

## Built on by (most cited first, 2 of 2)

- [Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Review](https://mlsanity.com/paper/2503.04797/) (2025)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://mlsanity.com/paper/2510.24446/) (2025)

## Abstract (arXiv)

Machine Translation (MT) Quality Estimation (QE) assesses translation reliability without reference texts. This study introduces "textual similarity" as a new metric for QE, using sentence transformers and cosine similarity to measure semantic closeness. Analyzing data from the MLQE-PE dataset, we found that textual similarity exhibits stronger correlations with human scores than traditional metrics (hter, model evaluation, sentence probability etc.). Employing GAMMs as a statistical tool, we demonstrated that textual similarity consistently outperforms other metrics across multiple language pairs in predicting human scores. We also found that "hter" actually failed to predict human scores in QE. Our findings highlight the effectiveness of textual similarity as a robust QE metric, recommending its integration with other metrics into QE frameworks and MT system training for improved accuracy and usability.