# Model-Agnostic Interpretability of Machine Learning

arXiv: https://arxiv.org/abs/1606.05386 · HTML: https://mlsanity.com/paper/1606.05386/

## Key facts

- Cited by 219 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2016.

## Built on by (most cited first, 30 of 219)

- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism](https://mlsanity.com/paper/2201.12987/) (2022)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Generative Causal Explanations for Graph Neural Networks](https://mlsanity.com/paper/2104.06643/) (2021)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations](https://mlsanity.com/paper/2207.04154/) (2022)
- [Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs](https://mlsanity.com/paper/2101.09824/) (2021)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Entropy-based Logic Explanations of Neural Networks](https://mlsanity.com/paper/2106.06804/) (2021)
- [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023)
- [Intriguing Properties of Data Attribution on Diffusion Models](https://mlsanity.com/paper/2311.00500/) (2023)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Logic Explained Networks](https://mlsanity.com/paper/2108.05149/) (2021)
- [EEG based Emotion Recognition: A Tutorial and Review](https://mlsanity.com/paper/2203.11279/) (2022)
- [Evaluating Robustness of Counterfactual Explanations](https://mlsanity.com/paper/2103.02354/) (2021)
- [The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations](https://mlsanity.com/paper/2205.03295/) (2022)
- [Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations](https://mlsanity.com/paper/2205.07277/) (2022)
- [On Efficiently Explaining Graph-Based Classifiers](https://mlsanity.com/paper/2106.01350/) (2021)
- [A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts](https://mlsanity.com/paper/2105.00290/) (2021)
- [DecompX: Explaining Transformers Decisions by Propagating Token Decomposition](https://mlsanity.com/paper/2306.02873/) (2023)
- [Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing](https://mlsanity.com/paper/2108.04990/) (2021)
- [The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations](https://mlsanity.com/paper/2212.11737/) (2022)
- [Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making](https://mlsanity.com/paper/2204.06916/) (2022)
- [To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods](https://mlsanity.com/paper/2106.00461/) (2021)
- [Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process](https://mlsanity.com/paper/2109.01433/) (2021)

## Abstract (arXiv)

Understanding why machine learning models behave the way they do empowers both system designers and end-users in many ways: in model selection, feature engineering, in order to trust and act upon the predictions, and in more intuitive user interfaces. Thus, interpretability has become a vital concern in machine learning, and work in the area of interpretable models has found renewed interest. In some applications, such models are as accurate as non-interpretable ones, and thus are preferred for their transparency. Even when they are not accurate, they may still be preferred when interpretability is of paramount importance. However, restricting machine learning to interpretable models is often a severe limitation. In this paper we argue for explaining machine learning predictions using model-agnostic approaches. By treating the machine learning models as black-box functions, these approaches provide crucial flexibility in the choice of models, explanations, and representations, improving debugging, comparison, and interfaces for a variety of users and models. We also outline the main challenges for such methods, and review a recently-introduced model-agnostic explanation approach (LIME) that addresses these challenges.