mlsanityData to 7 Oct 2026

Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings

arXiv 2109.07833 ↗First posted 16 Sep 2021cs.CL cs.HCcited by 9

Not on the map. Why?

Abstract (arXiv)

Natural language inference (NLI) requires models to learn and apply commonsense knowledge. These reasoning abilities are particularly important for explainable NLI systems that generate a natural language explanation in addition to their label prediction. The integration of external knowledge has been shown to improve NLI systems, here we investigate whether it can also improve their explanation capabilities. For this, we investigate different sources of external knowledge and evaluate the performance of our models on in-domain data as well as on special transfer datasets that are designed to assess fine-grained reasoning capabilities. We find that different sources of knowledge have a different effect on reasoning abilities, for example, implicit knowledge stored in language models can hinder reasoning on numbers and negations. Finally, we conduct the largest and most fine-grained explainable NLI crowdsourcing study to date. It reveals that even large differences in automatic performance scores do neither reflect in human ratings of label, explanation, commonsense nor grammar correctness.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

34 of 46 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  12. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
22 more
  1. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  2. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  3. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  4. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  5. 2019Abductive Commonsense Reasoningcited by 300
  6. 2016Rationalizing Neural Predictionscited by 281
  7. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  8. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  9. 2016Enhanced LSTM for Natural Language Inferencecited by 199
  10. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  11. 2020Generating Fact Checking Explanationscited by 132
  12. 2020NILE : Natural Language Inference with Faithful Natural Language Explanationscited by 96
  13. 2019Augmenting Neural Networks with First-order Logiccited by 51
  14. 2018Improving Natural Language Inference Using External Knowledge in the Science Questions Domaincited by 48
  15. 2017Neural Natural Language Inference Models Enhanced with External Knowledgecited by 40
  16. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  17. 2020Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transportcited by 20
  18. 2019Improving Question Answering with External Knowledgecited by 16
  19. 2020F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answeringcited by 11
  20. 2021KI-BERT: Infusing Knowledge Context for Better Language and Domain Understandingcited by 11
  21. 2019Several Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inferencecited by 6
  22. 2019Knowledge Enhanced Attention for Robust Natural Language Inferencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 1; 2024: 1; 2025: 2; 2026: 1.

9 papers.

  1. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  2. 2022Does Self-Rationalization Improve Robustness to Spurious Correlations?cited by 9
  3. 2025Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarkscited by 5
  4. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  5. 2025Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarkscited by 3
  6. 2022Towards the Generation of Musical Explanations with GPT-3cited by 1
  7. 2022Challenges in Explanation Quality Evaluationcited by 1
  8. 2026Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMscited by 1
  9. 2024Demystifying Faulty Code with LLM: Step-by-Step Reasoning for Explainable Fault Localizationcited by 0