# Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings

arXiv: https://arxiv.org/abs/2109.07833 · HTML: https://mlsanity.com/paper/2109.07833/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [WT5?! Training Text-to-Text Models to Explain their Predictions](https://mlsanity.com/paper/2004.14546/) (2020)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [NILE : Natural Language Inference with Faithful Natural Language Explanations](https://mlsanity.com/paper/2005.12116/) (2020)
- [Augmenting Neural Networks with First-order Logic](https://mlsanity.com/paper/1906.06298/) (2019)
- [Improving Natural Language Inference Using External Knowledge in the Science Questions Domain](https://mlsanity.com/paper/1809.05724/) (2018)
- [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017)
- [A Study of Automatic Metrics for the Evaluation of Natural Language Explanations](https://mlsanity.com/paper/2103.08545/) (2021)
- [Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transport](https://mlsanity.com/paper/2005.13111/) (2020)
- [Improving Question Answering with External Knowledge](https://mlsanity.com/paper/1902.00993/) (2019)

## Built on by (most cited first, 9 of 9)

- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [Does Self-Rationalization Improve Robustness to Spurious Correlations?](https://mlsanity.com/paper/2210.13575/) (2022)
- [Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks](https://mlsanity.com/paper/2510.01232/) (2025)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks](https://mlsanity.com/paper/2502.18339/) (2025)
- [Towards the Generation of Musical Explanations with GPT-3](https://mlsanity.com/paper/2206.08264/) (2022)
- [Challenges in Explanation Quality Evaluation](https://mlsanity.com/paper/2210.07126/) (2022)
- [Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs](https://mlsanity.com/paper/2605.29928/) (2026)
- [Demystifying Faulty Code with LLM: Step-by-Step Reasoning for Explainable Fault Localization](https://mlsanity.com/paper/2403.10507/) (2024)

## Abstract (arXiv)

Natural language inference (NLI) requires models to learn and apply commonsense knowledge. These reasoning abilities are particularly important for explainable NLI systems that generate a natural language explanation in addition to their label prediction. The integration of external knowledge has been shown to improve NLI systems, here we investigate whether it can also improve their explanation capabilities. For this, we investigate different sources of external knowledge and evaluate the performance of our models on in-domain data as well as on special transfer datasets that are designed to assess fine-grained reasoning capabilities. We find that different sources of knowledge have a different effect on reasoning abilities, for example, implicit knowledge stored in language models can hinder reasoning on numbers and negations. Finally, we conduct the largest and most fine-grained explainable NLI crowdsourcing study to date. It reveals that even large differences in automatic performance scores do neither reflect in human ratings of label, explanation, commonsense nor grammar correctness.