# Assessing the Local Interpretability of Machine Learning Models

arXiv: https://arxiv.org/abs/1902.03501 · HTML: https://mlsanity.com/paper/1902.03501/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2019.

## Built on by (most cited first, 23 of 23)

- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Rethinking Explainability as a Dialogue: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01875/) (2022)
- [Explainable Artificial Intelligence Approaches: A Survey](https://mlsanity.com/paper/2101.09429/) (2021)
- [Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks](https://mlsanity.com/paper/2212.05946/) (2022)
- [Selective Explanations: Leveraging Human Input to Align Explainable AI](https://mlsanity.com/paper/2301.09656/) (2023)
- [A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?](https://mlsanity.com/paper/2304.04780/) (2023)
- [Machine Explanations and Human Understanding](https://mlsanity.com/paper/2202.04092/) (2022)
- [Evaluating the Interpretability of Generative Models by Interactive Reconstruction](https://mlsanity.com/paper/2102.01264/) (2021)
- [On the Interaction of Belief Bias and Explanations](https://mlsanity.com/paper/2106.15355/) (2021)
- [Q-SENN: Quantized Self-Explaining Neural Networks](https://mlsanity.com/paper/2312.13839/) (2023)
- [Take 5: Interpretable Image Classification with a Handful of Features](https://mlsanity.com/paper/2303.13166/) (2023)
- [Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacy](https://mlsanity.com/paper/2402.07632/) (2024)
- [A Review of Explainable Artificial Intelligence in Manufacturing](https://mlsanity.com/paper/2107.02295/) (2021)
- [Interpretation Quality Score for Measuring the Quality of interpretability methods](https://mlsanity.com/paper/2205.12254/) (2022)
- [OpenHEXAI: An Open-Source Framework for Human-Centered Evaluation of Explainable Machine Learning](https://mlsanity.com/paper/2403.05565/) (2024)
- [Refusal as Silence: Gendered Disparities in Vision-Language Model Responses](https://mlsanity.com/paper/2406.08222/) (2024)
- [A Human-In-The-Loop Approach for Improving Fairness in Predictive Business Process Monitoring](https://mlsanity.com/paper/2508.17477/) (2025)
- [ACT: Agentic Classification Tree](https://mlsanity.com/paper/2509.26433/) (2025)
- [Opportunities for Machine Learning to Accelerate Halide Perovskite Commercialization and Scale-Up](https://mlsanity.com/paper/2110.03923/) (2021)
- [An Empirical Examination of the Evaluative AI Framework](https://mlsanity.com/paper/2411.08583/) (2024)
- [Empowering Decision Trees via Shape Function Branching](https://mlsanity.com/paper/2510.19040/) (2025)
- [Rules or Weights? Comparing User Understanding of Explainable AI Techniques with the Cognitive XAI-Adaptive Model](https://mlsanity.com/paper/2602.19620/) (2026)

## Abstract (arXiv)

The increasing adoption of machine learning tools has led to calls for accountability via model interpretability. But what does it mean for a machine learning model to be interpretable by humans, and how can this be assessed? We focus on two definitions of interpretability that have been introduced in the machine learning literature: simulatability (a user's ability to run a model on a given input) and "what if" local explainability (a user's ability to correctly determine a model's prediction under local changes to the input, given knowledge of the model's original prediction). Through a user study with 1,000 participants, we test whether humans perform well on tasks that mimic the definitions of simulatability and "what if" local explainability on models that are typically considered locally interpretable. To track the relative interpretability of models, we employ a simple metric, the runtime operation count on the simulatability task. We find evidence that as the number of operations increases, participant accuracy on the local interpretability tasks decreases. In addition, this evidence is consistent with the common intuition that decision trees and logistic regression models are interpretable and are more interpretable than neural networks.