# An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability

arXiv: https://arxiv.org/abs/2109.05327 · HTML: https://mlsanity.com/paper/2109.05327/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 106 references that are in the map.
- First posted on arXiv 11 Sep 2021.

## Builds on (most cited first, 30 of 32)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives](https://mlsanity.com/paper/1802.07623/) (2018)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI](https://mlsanity.com/paper/2003.07258/) (2020)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques](https://mlsanity.com/paper/2103.01035/) (2021)
- [On quantitative aspects of model interpretability](https://mlsanity.com/paper/2007.07584/) (2020)
- [Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations](https://mlsanity.com/paper/1912.09024/) (2019)
- [Interpretable & Explorable Approximations of Black Box Models](https://mlsanity.com/paper/1707.01154/) (2017)
- [Large-Scale QA-SRL Parsing](https://mlsanity.com/paper/1805.05377/) (2018)
- [LAReQA: Language-agnostic answer retrieval from a multilingual pool](https://mlsanity.com/paper/2004.05484/) (2020)
- [QADiscourse -- Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines](https://mlsanity.com/paper/2010.02815/) (2020)
- [MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models](https://mlsanity.com/paper/2005.02507/) (2020)
- [A Grounded Interaction Protocol for Explainable Artificial Intelligence](https://mlsanity.com/paper/1903.02409/) (2019)
- [Crowdsourcing Question-Answer Meaning Representations](https://mlsanity.com/paper/1711.05885/) (2017)
- [Modelling GDPR-Compliant Explanations for Trustworthy AI](https://mlsanity.com/paper/2109.04165/) (2021)
- [From Philosophy to Interfaces: an Explanatory Method and a Tool Inspired by Achinstein's Theory of Explanation](https://mlsanity.com/paper/2109.04171/) (2021)

## Built on by (most cited first, 16 of 16)

- [FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare](https://mlsanity.com/paper/2309.12325/) (2023)
- [Human-Centric Artificial Intelligence Architecture for Industry 5.0 Applications](https://mlsanity.com/paper/2203.10794/) (2022)
- [A Survey on Methods and Metrics for the Assessment of Explainability under the Proposed AI Act](https://mlsanity.com/paper/2110.11168/) (2021)
- [An Empirical Study on Compliance with Ranking Transparency in the Software Documentation of EU Online Platforms](https://mlsanity.com/paper/2312.14794/) (2023)
- [Enriching Artificial Intelligence Explanations with Knowledge Fragments](https://mlsanity.com/paper/2204.05579/) (2022)
- [Understanding User Preferences in Explainable Artificial Intelligence: A Survey and a Mapping Function Proposal](https://mlsanity.com/paper/2302.03180/) (2023)
- [XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensics](https://mlsanity.com/paper/2402.02452/) (2024)
- [Heterogeneous Subgraph Network with Prompt Learning for Interpretable Depression Detection on Social Media](https://mlsanity.com/paper/2407.09019/) (2024)
- [Case-based Explainability for Random Forest: Prototypes, Critics, Counter-factuals and Semi-factuals](https://mlsanity.com/paper/2408.06679/) (2024)
- [Aligning XAI with EU Regulations for Smart Biomedical Devices: A Methodology for Compliance Analysis](https://mlsanity.com/paper/2408.15121/) (2024)
- [L-XAIDS: A LIME-based eXplainable AI framework for Intrusion Detection Systems](https://mlsanity.com/paper/2508.17244/) (2025)
- [Computational Compliance for AI Regulation: Blueprint for a New Research Domain](https://mlsanity.com/paper/2601.04474/) (2026)
- [Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2604.06211/) (2026)
- [Who Benefits from AI Explanations? Towards Accessible and Interpretable Systems](https://mlsanity.com/paper/2508.10806/) (2025)
- [Security and Privacy in Agentic AI: Grand Challenges and Future Directions](https://mlsanity.com/paper/2607.06608/) (2026)
- [Integrating Feature Understandability into Feature Selection for Human-Centred Tabular Explanations](https://mlsanity.com/paper/2610.16627/) (2026)

## Abstract (arXiv)

Explainable AI was born as a pathway to allow humans to explore and understand the inner working of complex systems. However, establishing what is an explanation and objectively evaluating explainability are not trivial tasks. This paper presents a new model-agnostic metric to measure the Degree of Explainability of information in an objective way. We exploit a specific theoretical model from Ordinary Language Philosophy called the Achinstein's Theory of Explanations, implemented with an algorithm relying on deep language models for knowledge graph extraction and information retrieval. To understand whether this metric can measure explainability, we devised a few experiments and user studies involving more than 190 participants, evaluating two realistic systems for healthcare and finance using famous AI technology, including Artificial Neural Networks and TreeSHAP. The results we obtained are statistically significant (with P values lower than .01), suggesting that our proposed metric for measuring the Degree of Explainability is robust in several scenarios, and it aligns with concrete expectations.