# Designing Inherently Interpretable Machine Learning Models

arXiv: https://arxiv.org/abs/2111.01743 · HTML: https://mlsanity.com/paper/2111.01743/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 2 Nov 2021.

## Builds on (most cited first, 8 of 8)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020)
- [GAMI-Net: An Explainable Neural Network based on Generalized Additive Models with Structured Interactions](https://mlsanity.com/paper/2003.07132/) (2020)
- [Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification](https://mlsanity.com/paper/2011.04041/) (2020)
- [Enhancing Explainability of Neural Networks through Architecture Constraints](https://mlsanity.com/paper/1901.03838/) (2019)

## Built on by (most cited first, 23 of 23)

- [Explainable, Interpretable & Trustworthy AI for Intelligent Digital Twin: Case Study on Remaining Useful Life](https://mlsanity.com/paper/2301.06676/) (2023)
- [Monotonic Neural Additive Models: Pursuing Regulated Machine Learning Models for Credit Scoring](https://mlsanity.com/paper/2209.10070/) (2022)
- [A Comprehensive Review on Financial Explainable AI](https://mlsanity.com/paper/2309.11960/) (2023)
- [Interpreting and generalizing deep learning in physics-based problems with functional linear models](https://mlsanity.com/paper/2307.04569/) (2023)
- [PiML Toolbox for Interpretable Machine Learning Model Development and Diagnostics](https://mlsanity.com/paper/2305.04214/) (2023)
- [Interpretable Machine Learning based on Functional ANOVA Framework: Algorithms and Comparisons](https://mlsanity.com/paper/2305.15670/) (2023)
- [Space-scale Exploration of the Poor Reliability of Deep Learning Models: the Case of the Remote Sensing of Rooftop Photovoltaic Systems](https://mlsanity.com/paper/2408.07828/) (2024)
- [On marginal feature attributions of tree-based models](https://mlsanity.com/paper/2302.08434/) (2023)
- [Semantics, Ontology and Explanation](https://mlsanity.com/paper/2304.11124/) (2023)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Less Discriminatory Alternative and Interpretable XGBoost Framework for Binary Classification](https://mlsanity.com/paper/2410.19067/) (2024)
- [Inherently Interpretable Tree Ensemble Learning](https://mlsanity.com/paper/2410.19098/) (2024)
- [Interpretable (not just posthoc-explainable) medical claims modeling for discharge placement to prevent avoidable all-cause readmissions or death](https://mlsanity.com/paper/2208.12814/) (2022)
- [Interpretable (not just posthoc-explainable) heterogeneous survivor bias-corrected treatment effects for assignment of postdischarge interventions to prevent readmissions](https://mlsanity.com/paper/2304.09981/) (2023)
- [Wrapper Boxes: Faithful Attribution of Model Predictions to Training Data](https://mlsanity.com/paper/2311.08644/) (2023)
- [Are Logistic Models Really Interpretable?](https://mlsanity.com/paper/2406.13427/) (2024)
- [CHILLI: A data context-aware perturbation method for XAI](https://mlsanity.com/paper/2407.07521/) (2024)
- [Autoencoded sparse Bayesian in-IRT factorization, calibration, and amortized inference for the Work Disability Functional Assessment Battery](https://mlsanity.com/paper/2210.10952/) (2022)
- [A Comparison of Modeling Preprocessing Techniques](https://mlsanity.com/paper/2302.12042/) (2023)
- [Sound Explanation for Trustworthy Machine Learning](https://mlsanity.com/paper/2306.06134/) (2023)
- [Opening the Black Box: Towards inherently interpretable energy data imputation models using building physics insight](https://mlsanity.com/paper/2311.16632/) (2023)
- [Explainable post-training bias mitigation with distribution-based fairness metrics](https://mlsanity.com/paper/2504.01223/) (2025)
- [Sparse POD Mode Selection and Manifold Dimensionality Reduction with Neural Networks](https://mlsanity.com/paper/2605.27756/) (2026)

## Abstract (arXiv)

Interpretable machine learning (IML) becomes increasingly important in highly regulated industry sectors related to the health and safety or fundamental rights of human beings. In general, the inherently IML models should be adopted because of their transparency and explainability, while black-box models with model-agnostic explainability can be more difficult to defend under regulatory scrutiny. For assessing inherent interpretability of a machine learning model, we propose a qualitative template based on feature effects and model architecture constraints. It provides the design principles for high-performance IML model development, with examples given by reviewing our recent works on ExNN, GAMI-Net, SIMTree, and the Aletheia toolkit for local linear interpretability of deep ReLU networks. We further demonstrate how to design an interpretable ReLU DNN model with evaluation of conceptual soundness for a real case study of predicting credit default in home lending. We hope that this work will provide a practical guide of developing inherently IML models in high risk applications in banking industry, as well as other sectors.