mlsanityData to 14 Oct 2026

ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics

arXiv 2210.15615 ↗First posted 27 Oct 2022cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 34 of its 89 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

34 of 89 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  9. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
22 more
  1. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  2. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  3. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  4. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  6. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  8. 2019Evaluating Gender Bias in Machine Translationcited by 249
  9. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  10. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  11. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  12. 2017A Challenge Set Approach to Evaluating Machine Translationcited by 37
  13. 2022Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translationcited by 28
  14. 2020Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Taskcited by 25
  15. 2021NoiseQA: Challenge Set Evaluation for User-Centric Question Answeringcited by 25
  16. 2016Neural versus Phrase-Based Machine Translation Quality: a Case Studycited by 19
  17. 2022As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioningcited by 11
  18. 2017A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directionscited by 10
  19. 2022Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Taskcited by 7
  20. 2018Non-entailed subsequences as a challenge for natural language inferencecited by 5
  21. 2019Fine-grained evaluation of Quality Estimation for Machine translation based on a linguistically-motivated Test Suitecited by 2
  22. 2017Improving Discourse Relation Projection to Build Discourse Annotated Corporacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 7; 2025: 7; 2026: 3.

25 papers.

Sort:

By area:

  1. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  2. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  3. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
  4. 2024Machine Translation Meta Evaluation through Translation Accuracy Challenge Setscited by 18
  5. 2024Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Modelscited by 17
  6. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  7. 2022Extrinsic Evaluation of Machine Translation Metricscited by 14
  8. 2024MT-Ranker: Reference-free machine translation evaluation by inter-system rankingcited by 14
  9. 2024Pitfalls and Outlooks in Using COMETcited by 10
  10. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modelingcited by 9
  11. 2025QE4PE: Word-level Quality Estimation for Human Post-Editingcited by 6
  12. 2023BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluationcited by 5
13 more
  1. 2024TIGQA:An Expert Annotated Question Answering Dataset in Tigrinyacited by 5
  2. 2023Towards Fine-Grained Information: Identifying the Type and Location of Translation Errorscited by 4
  3. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  4. 2023ACES: Translation Accuracy Challenge Sets at WMT 2023cited by 3
  5. 2024Reference-based Metrics Disprove Themselves in Question Generationcited by 3
  6. 2025Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?cited by 2
  7. 2026Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translationscited by 2
  8. 2023Reranking for Natural Language Generation from Logical Forms: A Study based on Large Language Modelscited by 1
  9. 2025Generating Difficult-to-Translate Textscited by 1
  10. 2025Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotationscited by 1
  11. 2025Do Not Change Me: On Transferring Entities Without Modification in Neural Machine Translation -- a Multilingual Perspectivecited by 0
  12. 2026Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpuscited by 0
  13. 2026Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Studycited by 0