# A Challenge Set Approach to Evaluating Machine Translation

arXiv: https://arxiv.org/abs/1704.07431 · HTML: https://mlsanity.com/paper/1704.07431/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2017.

## Built on by (most cited first, 30 of 37)

- [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021)
- [Democratizing Neural Machine Translation with OPUS-MT](https://mlsanity.com/paper/2212.01936/) (2022)
- [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022)
- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [NoiseQA: Challenge Set Evaluation for User-Centric Question Answering](https://mlsanity.com/paper/2102.08345/) (2021)
- [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://mlsanity.com/paper/2210.15615/) (2022)
- [Deep Learning on a Healthy Data Diet: Finding Important Examples for Fairness](https://mlsanity.com/paper/2211.11109/) (2022)
- [Investigating Failures of Automatic Translation in the Case of Unambiguous Gender](https://mlsanity.com/paper/2104.07838/) (2021)
- [A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models](https://mlsanity.com/paper/2305.01181/) (2023)
- [Angler: Helping Machine Translation Practitioners Prioritize Model Improvements](https://mlsanity.com/paper/2304.05967/) (2023)
- [Should We Attend More or Less? Modulating Attention for Fairness](https://mlsanity.com/paper/2305.13088/) (2023)
- [PreQuEL: Quality Estimation of Machine Translation Outputs in Advance](https://mlsanity.com/paper/2205.09178/) (2022)
- [Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement](https://mlsanity.com/paper/2212.04523/) (2022)
- [Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus](https://mlsanity.com/paper/2305.11142/) (2023)
- [Quality and Quantity of Machine Translation References for Automatic Metrics](https://mlsanity.com/paper/2401.01283/) (2024)
- [Translating the Unseen? Yoruba-English MT in Low-Resource, Morphologically-Unmarked Settings](https://mlsanity.com/paper/2103.04225/) (2021)
- [Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biases](https://mlsanity.com/paper/2305.18917/) (2023)
- [A Bilingual Parallel Corpus with Discourse Annotations](https://mlsanity.com/paper/2210.14667/) (2022)
- [Sometimes We Want Translationese](https://mlsanity.com/paper/2104.07623/) (2021)
- [ACES: Translation Accuracy Challenge Sets at WMT 2023](https://mlsanity.com/paper/2311.01153/) (2023)
- [GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection?](https://mlsanity.com/paper/2402.15238/) (2024)
- [Tribrid: Stance Classification with Neural Inconsistency Detection](https://mlsanity.com/paper/2109.06508/) (2021)
- [Identifying and Benchmarking Natural Out-of-Context Prediction Problems](https://mlsanity.com/paper/2110.13223/) (2021)
- [Appropriateness is all you need!](https://mlsanity.com/paper/2304.14553/) (2023)
- [A Multilingual Evaluation of NER Robustness to Adversarial Inputs](https://mlsanity.com/paper/2305.18933/) (2023)
- [Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss](https://mlsanity.com/paper/2604.12911/) (2026)
- [Part of Speech and Universal Dependency effects on English Arabic Machine Translation](https://mlsanity.com/paper/2106.00745/) (2021)
- [Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets](https://mlsanity.com/paper/2205.06871/) (2022)
- [The Comparison of Translationese in Machine Translation and Human Transation in terms of Translation Relations](https://mlsanity.com/paper/2404.08661/) (2024)
- [Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution](https://mlsanity.com/paper/2410.09448/) (2024)

## Abstract (arXiv)

Neural machine translation represents an exciting leap forward in translation quality. But what longstanding weaknesses does it resolve, and which remain? We address these questions with a challenge set approach to translation evaluation and error analysis. A challenge set consists of a small set of sentences, each hand-designed to probe a system's capacity to bridge a particular structural divergence between languages. To exemplify this approach, we present an English-French challenge set, and use it to analyze phrase-based and neural systems. The resulting analysis provides not only a more fine-grained picture of the strengths of neural systems, but also insight into which linguistic phenomena remain out of reach.