mlsanityData to 7 Oct 2026

A Study of Automatic Metrics for the Evaluation of Natural Language Explanations

arXiv 2103.08545 ↗First posted 15 Mar 2021cs.CL cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

As transparency becomes key for robotics and AI, it will be necessary to evaluate the methods through which transparency is provided, including automatically generated natural language (NL) explanations. Here, we explore parallels between the generation of such explanations and the much-studied field of evaluation of Natural Language Generation (NLG). Specifically, we investigate which of the NLG evaluation measures map well to explanations. We present the ExBAN corpus: a crowd-sourced corpus of NL explanations for Bayesian Networks. We run correlations comparing human subjective ratings with NLG automatic measures. We find that embedding-based automatic NLG evaluation methods, such as BERTScore and BLEURT, have a higher correlation with human ratings, compared to word-overlap metrics, such as BLEU and ROUGE. This work has implications for Explainable AI and transparent robotic and autonomous systems.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 32 of its 85 references that are in the map. First posted on arXiv 15 Mar 2021.

Builds on

32 of 85 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  9. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  12. 2019ELI5: Long Form Question Answeringcited by 567
20 more
  1. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  2. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  3. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  4. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  5. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  6. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  7. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  8. 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
  9. 2019Text Generation from Knowledge Graphs with Graph Transformerscited by 122
  10. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  11. 2020NILE : Natural Language Inference with Faithful Natural Language Explanationscited by 96
  12. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  13. 2017Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciencescited by 84
  14. 2018Model Agnostic Supervised Local Explanationscited by 83
  15. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  16. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60
  17. 2020QED: A Framework and Dataset for Explanations in Question Answeringcited by 55
  18. 2019Measurable Counterfactual Local Explanations for Any Classifiercited by 48
  19. 2019A Grounded Interaction Protocol for Explainable Artificial Intelligencecited by 22
  20. 2014Most Relevant Explanation in Bayesian Networkscited by 3

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 11; 2023: 8; 2024: 9; 2025: 2.

34 papers.

Sort:

By area:

  1. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  2. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  3. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  4. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  5. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  6. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 76
  7. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  8. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  9. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  10. 2023SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluationcited by 27
  11. 2022Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speechcited by 25
  12. 2021Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfercited by 23
22 more
  1. 2022ExPUNations: Augmenting Puns with Keywords and Explanationscited by 21
  2. 2024ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?cited by 17
  3. 2024Can We Verify Step by Step for Incorrect Answer Detection?cited by 15
  4. 2022Investigating the Benefits of Free-Form Rationalescited by 13
  5. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  6. 2021Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratingscited by 9
  7. 2023Characterizing Large Language Models as Rationalizers of Knowledge-intensive Taskscited by 9
  8. 2024Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandemcited by 8
  9. 2022Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanationscited by 7
  10. 2023Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AIcited by 3
  11. 2025Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarkscited by 3
  12. 2022On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalizationcited by 2
  13. 2022Towards Human-Centred Explainability Benchmarks For Text Classificationcited by 2
  14. 2023Towards Explainable and Safe Conversational Agents for Mental Health: A Surveycited by 2
  15. 2023Situated Natural Language Explanationscited by 2
  16. 2023Is Explanation the Cure? Misinformation Mitigation in the Short Term and Long Termcited by 2
  17. 2022Scientific Explanation and Natural Language: A Unified Epistemological-Linguistic Perspective for Explainable AIcited by 1
  18. 2022Challenges in Explanation Quality Evaluationcited by 1
  19. 2023Black-box Attacks on Image Activity Prediction and its Natural Language Explanationscited by 1
  20. 2024Scenarios and Approaches for Situated Natural Language Explanationscited by 1
  21. 2024EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correctioncited by 1
  22. 2024Automated Trustworthiness Testing for Machine Learning Classifierscited by 0