mlsanityData to 14 Oct 2026

The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics

arXiv 2305.11806 ↗First posted 19 May 2023cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU. Yet, neural metrics are, to a great extent, "black boxes" returning a single sentence-level score without transparency about the decision-making process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at: https://github.com/Unbabel/COMET/tree/explainable-metrics.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

29 of 52 references.

References · 29

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  4. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2019Attention is not Explanationcited by 929
  8. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 207
  9. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  10. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 159
  11. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  12. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
17 more
  1. 2022The Geometry of Multilingual Language Model Representationscited by 95
  2. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  3. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  4. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 61
  5. 2020TransQuest: Translation Quality Estimation with Cross-lingual Transformerscited by 59
  6. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  7. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  8. 2019Evaluating Recurrent Neural Network Explanationscited by 29
  9. 2022UniTE: Unified Translation Evaluationcited by 28
  10. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  11. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  12. 2022SALTED: A Framework for SAlient Long-Tail Translation Error Detectioncited by 23
  13. 2021Translation Error Detection as Rationale Extractioncited by 22
  14. 2022Learning to Scaffold: Optimizing Model Explanations for Teachingcited by 20
  15. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  16. 2022RoBLEURT Submission for the WMT2021 Metrics Taskcited by 9
  17. 2022A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inferencecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 6; 2025: 1; 2026: 3.

15 papers.

Sort:

  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  2. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  3. 2024xTower: A Multilingual LLM for Explaining and Correcting Translation Errorscited by 17
  4. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  5. 2024Pitfalls and Outlooks in Using COMETcited by 10
  6. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  7. 2024Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translationcited by 6
  8. 2023Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documentscited by 5
  9. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  10. 2024Aligning Translation-Specific Understanding to General Understanding in Large Language Modelscited by 3
  11. 2026Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translationcited by 1
  12. 2023Trained MT Metrics Learn to Cope with Machine-translated Referencescited by 0
3 more
  1. 2024Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Studycited by 0
  2. 2026Fluency and Faithfulness in Human and Machine Literary Translationcited by 0
  3. 2026Source-Free MT Evaluation Is Not MT Evaluationcited by 0