# Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations

arXiv: https://arxiv.org/abs/1912.09024 · HTML: https://mlsanity.com/paper/1912.09024/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2019.

## Built on by (most cited first, 30 of 64)

- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation](https://mlsanity.com/paper/2305.02231/) (2023)
- [Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications](https://mlsanity.com/paper/2103.04244/) (2021)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant](https://mlsanity.com/paper/2501.17546/) (2025)
- [Explainable AI for clinical and remote health applications: a survey on tabular and time series data](https://mlsanity.com/paper/2209.06528/) (2022)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [Deep ROC Analysis and AUC as Balanced Average Accuracy to Improve Model Selection, Understanding and Interpretation](https://mlsanity.com/paper/2103.11357/) (2021)
- [An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability](https://mlsanity.com/paper/2109.05327/) (2021)
- [A Review of the Role of Causality in Developing Trustworthy AI Systems](https://mlsanity.com/paper/2302.06975/) (2023)
- [FUTURE-AI: Guiding Principles and Consensus Recommendations for Trustworthy Artificial Intelligence in Medical Imaging](https://mlsanity.com/paper/2109.09658/) (2021)
- [Resisting Out-of-Distribution Data Problem in Perturbation of XAI](https://mlsanity.com/paper/2107.14000/) (2021)
- [Greybox XAI: a Neural-Symbolic learning framework to produce interpretable predictions for image classification](https://mlsanity.com/paper/2209.14974/) (2022)
- [Keep Your Friends Close and Your Counterfactuals Closer: Improved Learning From Closest Rather Than Plausible Counterfactual Explanations in an Abstract Setting](https://mlsanity.com/paper/2205.05515/) (2022)
- [Explainable AI for Bioinformatics: Methods, Tools, and Applications](https://mlsanity.com/paper/2212.13261/) (2022)
- [Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition](https://mlsanity.com/paper/2305.18011/) (2023)
- [Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models](https://mlsanity.com/paper/2309.07149/) (2023)
- [Let's Go to the Alien Zoo: Introducing an Experimental Framework to Study Usability of Counterfactual Explanations for Machine Learning](https://mlsanity.com/paper/2205.03398/) (2022)
- [A Survey on Methods and Metrics for the Assessment of Explainability under the Proposed AI Act](https://mlsanity.com/paper/2110.11168/) (2021)
- [A Meta Survey of Quality Evaluation Criteria in Explanation Methods](https://mlsanity.com/paper/2203.13929/) (2022)
- [How well can a large language model explain business processes as perceived by users?](https://mlsanity.com/paper/2401.12846/) (2024)
- [Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance](https://mlsanity.com/paper/2207.01482/) (2022)
- [What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks](https://mlsanity.com/paper/2403.12730/) (2024)
- [NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systems](https://mlsanity.com/paper/2601.11004/) (2026)
- [Applications of Explainable AI for 6G: Technical Aspects, Use Cases, and Research Challenges](https://mlsanity.com/paper/2112.04698/) (2021)
- [Segmentation-guided Domain Adaptation and Data Harmonization of Multi-device Retinal Optical Coherence Tomography using Cycle-Consistent Generative Adversarial Networks](https://mlsanity.com/paper/2208.14635/) (2022)
- [AI, Opacity, and Personal Autonomy](https://mlsanity.com/paper/2210.08995/) (2022)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)

## Abstract (arXiv)

Recent success in Artificial Intelligence (AI) and Machine Learning (ML) allow problem solving automatically without any human intervention. Autonomous approaches can be very convenient. However, in certain domains, e.g., in the medical domain, it is necessary to enable a domain expert to understand, why an algorithm came up with a certain result. Consequently, the field of Explainable AI (xAI) rapidly gained interest worldwide in various domains, particularly in medicine. Explainable AI studies transparency and traceability of opaque AI/ML and there are already a huge variety of methods. For example with layer-wise relevance propagation relevant parts of inputs to, and representations in, a neural network which caused a result, can be highlighted. This is a first important step to ensure that end users, e.g., medical professionals, assume responsibility for decision making with AI/ML and of interest to professionals and regulators. Interactive ML adds the component of human expertise to AI/ML processes by enabling them to re-enact and retrace AI/ML results, e.g. let them check it for plausibility. This requires new human-AI interfaces for explainable AI. In order to build effective and efficient interactive human-AI interfaces we have to deal with the question of how to evaluate the quality of explanations given by an explainable AI system. In this paper we introduce our System Causability Scale (SCS) to measure the quality of explanations. It is based on our notion of Causability (Holzinger et al., 2019) combined with concepts adapted from a widely accepted usability scale.