# Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation

arXiv: https://arxiv.org/abs/2305.19330 · HTML: https://mlsanity.com/paper/2305.19330/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 67 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 25 of 25)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [BLEU might be Guilty but References are not Innocent](https://mlsanity.com/paper/2004.06063/) (2020)
- [Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation](https://mlsanity.com/paper/2105.08504/) (2021)
- [The Effect of Translationese in Machine Translation Test Sets](https://mlsanity.com/paper/1906.08069/) (2019)
- [Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET](https://mlsanity.com/paper/2202.05148/) (2022)
- [Translationese in Machine Translation Evaluation](https://mlsanity.com/paper/1906.09833/) (2019)
- [UniTE: Unified Translation Evaluation](https://mlsanity.com/paper/2204.13346/) (2022)
- [Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords](https://mlsanity.com/paper/2007.03006/) (2020)
- [Extrinsic Evaluation of Machine Translation Metrics](https://mlsanity.com/paper/2212.10297/) (2022)
- [RoBLEURT Submission for the WMT2021 Metrics Task](https://mlsanity.com/paper/2204.13352/) (2022)
- [Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation](https://mlsanity.com/paper/2109.10859/) (2021)
- [A Study of Genetic Algorithms for Hyperparameter Optimization of Neural Networks in Machine Translation](https://mlsanity.com/paper/2009.08928/) (2020)
- [CUNI Submission in WMT22 General Task](https://mlsanity.com/paper/2211.16174/) (2022)

## Built on by (most cited first, 5 of 5)

- [MOPO: Multi-Objective Prompt Optimization for Affective Text Generation](https://mlsanity.com/paper/2412.12948/) (2024)
- [Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation](https://mlsanity.com/paper/2407.12832/) (2024)
- [A Survey on the Real Power of ChatGPT](https://mlsanity.com/paper/2405.00704/) (2024)
- [Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?](https://mlsanity.com/paper/2409.09568/) (2024)
- [Cogs in a Machine, Doing What They're Meant to Do -- The AMI Submission to the WMT24 General Translation Task](https://mlsanity.com/paper/2410.03381/) (2024)

## Abstract (arXiv)

We propose a genetic algorithm (GA) based method for modifying n-best lists produced by a machine translation (MT) system. Our method offers an innovative approach to improving MT quality and identifying weaknesses in evaluation metrics. Using common GA operations (mutation and crossover) on a list of hypotheses in combination with a fitness function (an arbitrary MT metric), we obtain novel and diverse outputs with high metric scores. With a combination of multiple MT metrics as the fitness function, the proposed method leads to an increase in translation quality as measured by other held-out automatic metrics. With a single metric (including popular ones such as COMET) as the fitness function, we find blind spots and flaws in the metric. This allows for an automated search for adversarial examples in an arbitrary metric, without prior assumptions on the form of such example. As a demonstration of the method, we create datasets of adversarial examples and use them to show that reference-free COMET is substantially less robust than the reference-based version.