mlsanityData to 7 Oct 2026

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

arXiv 2103.11251 ↗First posted 20 Mar 2021cs.LG stat.MLcited by 364

Not on the map. Why?

Abstract (arXiv)

Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the "Rashomon set" of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.

Key facts. Cited by 364 papers in the map (data through 7 Oct 2026). Builds on 123 of its 347 references that are in the map. First posted on arXiv 20 Mar 2021.

Builds on

123 of 347 references.

References · 123

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2015Spatial Transformer Networkscited by 1,423
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  8. 2018Sanity Checks for Saliency Mapscited by 1,149
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2020Concept Bottleneck Modelscited by 1,022
  11. 2020Object-Centric Learning with Slot Attentioncited by 865
  12. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
38 more
  1. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  2. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  3. 2017Dynamic Routing Between Capsulescited by 689
  4. 2018Disentangling by Factorisingcited by 614
  5. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  7. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  8. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 364
  9. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  11. 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 318
  12. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  13. 2018Towards a Definition of Disentangled Representationscited by 294
  14. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  15. 2019Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Datacited by 261
  16. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  17. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 243
  18. 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
  19. 2018Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equationscited by 233
  20. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  21. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  22. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 213
  23. 2019Explainable Machine Learning in Deploymentcited by 210
  24. 2018Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learningcited by 203
  25. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  26. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  27. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  28. 2020Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualizationcited by 170
  29. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
  30. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  31. 2019Predictive Multiplicity in Classificationcited by 162
  32. 2017Interpretable Convolutional Neural Networkscited by 161
  33. 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 141
  34. 2015Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction modelcited by 140
  35. 2020Understanding and mitigating gradient pathologies in physics-informed neural networkscited by 134
  36. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  37. 2021How to represent part-whole hierarchies in a neural networkcited by 123
  38. 2015Deep Convolutional Inverse Graphics Networkcited by 121

50 most cited of 123 All 123 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 75; 2023: 79; 2024: 91; 2025: 61; 2026: 33.

364 papers.

Sort:

By area:

  1. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  2. 2022Visual Classification via Description from Large Language Modelscited by 331
  3. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  4. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  5. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  6. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  7. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  8. 2022Exploring the Whole Rashomon Set of Sparse Decision Treescited by 80
  9. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  10. 2022Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Offcited by 61
  11. 2022From Shapley Values to Generalized Additive Models and backcited by 59
  12. 2022Visual Recognition with Deep Nearest Centroidscited by 59
38 more
  1. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  2. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
  3. 2024Amazing Things Come From Having Many Good Modelscited by 52
  4. 2021Entropy-based Logic Explanations of Neural Networkscited by 51
  5. 2022Logic-Based Explainability in Machine Learningcited by 47
  6. 2022Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capabilitycited by 46
  7. 2021Local Explanations via Necessity and Sufficiency: Unifying Theory and Practicecited by 45
  8. 2021Logic Explained Networkscited by 45
  9. 2021A Framework for Learning Ante-hoc Explainable Models via Conceptscited by 45
  10. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  11. 2022On Tackling Explanation Redundancy in Decision Treescited by 44
  12. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  13. 2022Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Reviewcited by 37
  14. 2022Visual correspondence-based explanations improve AI robustness and human-AI team accuracycited by 37
  15. 2022Augmenting Interpretable Models with LLMs during Trainingcited by 36
  16. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  17. 2021Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologiescited by 34
  18. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  19. 2023ProtoSeg: Interpretable Semantic Segmentation with Prototypical Partscited by 33
  20. 2023A Path to Simpler Models Starts With Noisecited by 33
  21. 2024Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretationscited by 33
  22. 2021Adaptive wavelet distillation from neural networks through interpretationscited by 32
  23. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  24. 2022Interpretable by Design: Learning Predictors by Composing Interpretable Queriescited by 26
  25. 2022Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learningcited by 26
  26. 2024Interpretable Image Classification with Adaptive Prototype-based Vision Transformerscited by 25
  27. 2023ICICLE: Interpretable Class Incremental Continual Learningcited by 24
  28. 2022Computing Abductive Explanations for Boosted Treescited by 23
  29. 2023Dear XAI Community, We Need to Talk! Fundamental Misconceptions in Current XAI Researchcited by 22
  30. 2021Characterizing the risk of fairwashingcited by 21
  31. 2024Position: Explain to Question not to Justifycited by 20
  32. 2024Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Conceptscited by 20
  33. 2022Fast Sparse Classification for Generalized Linear and Additive Modelscited by 19
  34. 2021GAM Changer: Editing Generalized Additive Models with Interactive Visualizationcited by 18
  35. 2022Hierarchical Shrinkage: improving the accuracy and interpretability of tree-based methodscited by 18
  36. 2024Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discoverycited by 18
  37. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  38. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18

50 most cited of 364 All 364 (JSON)