# Evaluating Explanation Methods for Neural Machine Translation

arXiv: https://arxiv.org/abs/2005.01672 · HTML: https://mlsanity.com/paper/2005.01672/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 14 of 14)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations](https://mlsanity.com/paper/2307.08678/) (2023)
- [Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals](https://mlsanity.com/paper/2310.00603/) (2023)
- [Evaluating Saliency Methods for Neural Language Models](https://mlsanity.com/paper/2104.05824/) (2021)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution](https://mlsanity.com/paper/2106.01518/) (2021)
- [GraphNarrator: Generating Textual Explanations for Graph Neural Networks](https://mlsanity.com/paper/2410.15268/) (2024)
- [Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods](https://mlsanity.com/paper/2310.05619/) (2023)
- [On Synthetic Data for Back Translation](https://mlsanity.com/paper/2310.13675/) (2023)
- [Visualizing the Relationship Between Encoded Linguistic Information and Task Performance](https://mlsanity.com/paper/2203.15860/) (2022)
- [Evaluating Explanation Methods for Vision-and-Language Navigation](https://mlsanity.com/paper/2310.06654/) (2023)
- [Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillation](https://mlsanity.com/paper/2603.11342/) (2026)
- [Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness](https://mlsanity.com/paper/2605.27879/) (2026)

## Abstract (arXiv)

Recently many efforts have been devoted to interpreting the black-box NMT models, but little progress has been made on metrics to evaluate explanation methods. Word Alignment Error Rate can be used as such a metric that matches human understanding, however, it can not measure explanation methods on those target words that are not aligned to any source word. This paper thereby makes an initial attempt to evaluate explanation methods from an alternative viewpoint. To this end, it proposes a principled metric based on fidelity in regard to the predictive behavior of the NMT model. As the exact computation for this metric is intractable, we employ an efficient approach as its approximation. On six standard translation tasks, we quantitatively evaluate several explanation methods in terms of the proposed metric and we reveal some valuable findings for these explanation methods in our experiments.