# A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference

arXiv: https://arxiv.org/abs/2204.05428 · HTML: https://mlsanity.com/paper/2204.05428/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- ["What is Relevant in a Text Document?": An Interpretable Machine Learning Approach](https://mlsanity.com/paper/1612.07843/) (2016)
- [Interpretation of NLP models through input marginalization](https://mlsanity.com/paper/2010.13984/) (2020)
- [Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values](https://mlsanity.com/paper/1810.03307/) (2018)
- [Evaluating Saliency Methods for Neural Language Models](https://mlsanity.com/paper/2104.05824/) (2021)

## Built on by (most cited first, 5 of 5)

- [Explaining How Transformers Use Context to Build Predictions](https://mlsanity.com/paper/2305.12535/) (2023)
- [The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics](https://mlsanity.com/paper/2305.11806/) (2023)
- [Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions](https://mlsanity.com/paper/2305.10614/) (2023)
- [Faithfulness Measurable Masked Language Models](https://mlsanity.com/paper/2310.07819/) (2023)
- [X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs](https://mlsanity.com/paper/2309.08873/) (2023)

## Abstract (arXiv)

Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility. First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations.We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods. Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, to support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.