mlsanityData to 7 Oct 2026

A Challenge Set Approach to Evaluating Machine Translation

arXiv 1704.07431 ↗First posted 24 Apr 2017cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Neural machine translation represents an exciting leap forward in translation quality. But what longstanding weaknesses does it resolve, and which remain? We address these questions with a challenge set approach to translation evaluation and error analysis. A challenge set consists of a small set of sentences, each hand-designed to probe a system's capacity to bridge a particular structural divergence between languages. To exemplify this approach, we present an English-French challenge set, and use it to analyze phrase-based and neural systems. The resulting analysis provides not only a more fine-grained picture of the strengths of neural systems, but also insight into which linguistic phenomena remain out of reach.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 10; 2023: 8; 2024: 4; 2025: 3; 2026: 3.

37 papers.

Sort:

  1. 2021BlonDe: An Automatic Evaluation Metric for Document-level Machine Translationcited by 58
  2. 2022Democratizing Neural Machine Translation with OPUS-MTcited by 57
  3. 2022MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translationcited by 38
  4. 2022Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translationcited by 28
  5. 2021NoiseQA: Challenge Set Evaluation for User-Centric Question Answeringcited by 25
  6. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
  7. 2022Deep Learning on a Healthy Data Diet: Finding Important Examples for Fairnesscited by 21
  8. 2021Investigating Failures of Automatic Translation in the Case of Unambiguous Gendercited by 20
  9. 2023A Paradigm Shift: The Future of Machine Translation Lies with Large Language Modelscited by 19
  10. 2023Angler: Helping Machine Translation Practitioners Prioritize Model Improvementscited by 11
  11. 2023Should We Attend More or Less? Modulating Attention for Fairnesscited by 11
  12. 2022PreQuEL: Quality Estimation of Machine Translation Outputs in Advancecited by 9
25 more
  1. 2022Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreementcited by 7
  2. 2023Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpuscited by 7
  3. 2024Quality and Quantity of Machine Translation References for Automatic Metricscited by 7
  4. 2021Translating the Unseen? Yoruba-English MT in Low-Resource, Morphologically-Unmarked Settingscited by 5
  5. 2023Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biasescited by 5
  6. 2021Sometimes We Want Translationesecited by 3
  7. 2022A Bilingual Parallel Corpus with Discourse Annotationscited by 3
  8. 2023ACES: Translation Accuracy Challenge Sets at WMT 2023cited by 3
  9. 2024GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection?cited by 3
  10. 2021Tribrid: Stance Classification with Neural Inconsistency Detectioncited by 2
  11. 2021Identifying and Benchmarking Natural Out-of-Context Prediction Problemscited by 2
  12. 2023Appropriateness is all you need!cited by 2
  13. 2023A Multilingual Evaluation of NER Robustness to Adversarial Inputscited by 2
  14. 2021Part of Speech and Universal Dependency effects on English Arabic Machine Translationcited by 1
  15. 2022Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasetscited by 1
  16. 2024The Comparison of Translationese in Machine Translation and Human Transation in terms of Translation Relationscited by 1
  17. 2024Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolutioncited by 1
  18. 2025A Large-Scale Benchmark for Vietnamese Sentence Paraphrasescited by 1
  19. 2025From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modelingcited by 1
  20. 2025Generating Difficult-to-Translate Textscited by 1
  21. 2026Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Misscited by 1
  22. 2026Augmenting Text to Increase Translation Difficultycited by 1
  23. 2021Repairing Pronouns in Translation with BERT-Based Post-Editingcited by 0
  24. 2022HiJoNLP at SemEval-2022 Task 2: Detecting Idiomaticity of Multiword Expressions using Multilingual Pretrained Language Modelscited by 0
  25. 2026Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?cited by 0