# What is Interpretable? Using Machine Learning to Design Interpretable Decision-Support Systems

arXiv: https://arxiv.org/abs/1811.10799 · HTML: https://mlsanity.com/paper/1811.10799/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2018.

## Built on by (most cited first, 11 of 11)

- [Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties](https://mlsanity.com/paper/2103.08951/) (2021)
- [The Role of Explainability in Assuring Safety of Machine Learning in Healthcare](https://mlsanity.com/paper/2109.00520/) (2021)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [Toward Interpretable Sleep Stage Classification Using Cross-Modal Transformers](https://mlsanity.com/paper/2208.06991/) (2022)
- [Comprehensible Artificial Intelligence on Knowledge Graphs: A survey](https://mlsanity.com/paper/2404.03499/) (2024)
- [Towards Trustworthy Deception Detection: Benchmarking Model Robustness across Domains, Modalities, and Languages](https://mlsanity.com/paper/2104.11761/) (2021)
- [Self-explaining Neural Network with Concept-based Explanations for ICU Mortality Prediction](https://mlsanity.com/paper/2110.04598/) (2021)
- [A survey and taxonomy of methods interpreting random forest models](https://mlsanity.com/paper/2407.12759/) (2024)
- [Interpreting deep embeddings for disease progression clustering](https://mlsanity.com/paper/2307.06060/) (2023)
- [Optimising Human-AI Collaboration by Learning Convincing Explanations](https://mlsanity.com/paper/2311.07426/) (2023)
- [Two-step counterfactual generation for OOD examples](https://mlsanity.com/paper/2302.05196/) (2023)

## Abstract (arXiv)

Recent efforts in Machine Learning (ML) interpretability have focused on creating methods for explaining black-box ML models. However, these methods rely on the assumption that simple approximations, such as linear models or decision-trees, are inherently human-interpretable, which has not been empirically tested. Additionally, past efforts have focused exclusively on comprehension, neglecting to explore the trust component necessary to convince non-technical experts, such as clinicians, to utilize ML models in practice. In this paper, we posit that reinforcement learning (RL) can be used to learn what is interpretable to different users and, consequently, build their trust in ML models. To validate this idea, we first train a neural network to provide risk assessments for heart failure patients. We then design a RL-based clinical decision-support system (DSS) around the neural network model, which can learn from its interactions with users. We conduct an experiment involving a diverse set of clinicians from multiple institutions in three different countries. Our results demonstrate that ML experts cannot accurately predict which system outputs will maximize clinicians' confidence in the underlying neural network model, and suggest additional findings that have broad implications to the future of research into ML interpretability and the use of ML in medicine.