mlsanityData to 7 Oct 2026

Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks

arXiv 2311.00508 ↗First posted 1 Nov 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We investigate MT evaluation metric performance on adversarially-synthesized texts, to shed light on metric robustness. We experiment with word- and character-level attacks on three popular machine translation metrics: BERTScore, BLEURT, and COMET. Our human experiments validate that automatic metrics tend to overpenalize adversarially-degraded translations. We also identify inconsistencies in BERTScore ratings, where it judges the original sentence and the adversarially-degraded one as similar, while judging the degraded translation as notably worse than the original with respect to the reference. We identify patterns of brittleness that motivate more robust metric development.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

21 of 34 references.

References · 21

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  6. 2018Generating Natural Language Adversarial Examplescited by 422
  7. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  8. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  9. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  10. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  11. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 138
  12. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
9 more
  1. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  2. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  3. 2021Learning Compact Metrics for MTcited by 93
  4. 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
  5. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  6. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  7. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  8. 2021Multi-granularity Textual Adversarial Attack with Behavior Cloningcited by 21
  9. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2026Diagnosable ColBERT: Debugging Late-Interaction Retrieval Models Using a Learned Latent Space as Referencecited by 2
  2. 2025Q-FAKER: Query-free Hard Black-box Attack via Controlled Generationcited by 1
  3. 2024AUEB-Archimedes at RIRAG-2025: Is obligation concatenation really all you need?cited by 0
  4. 2025Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languagescited by 0
  5. 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservationcited by 0