mlsanityData to 7 Oct 2026

Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations

arXiv 2103.01890 ↗First posted 2 Mar 2021stat.ML cs.AI cs.CVcited by 59

Not on the map. Why?

Abstract (arXiv)

While the need for interpretable machine learning has been established, many common approaches are slow, lack fidelity, or hard to evaluate. Amortized explanation methods reduce the cost of providing interpretations by learning a global selector model that returns feature importances for a single instance of data. The selector model is trained to optimize the fidelity of the interpretations, as evaluated by a predictor model for the target. Popular methods learn the selector and predictor model in concert, which we show allows predictions to be encoded within interpretations. We introduce EVAL-X as a method to quantitatively evaluate interpretations and REAL-X as an amortized explanation method, which learn a predictor model that approximates the true data generating distribution given any subset of the input. We show EVAL-X can detect when predictions are encoded in interpretations and show the advantages of REAL-X through quantitative and radiologist evaluation.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

22 of 31 references.

References · 22

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  8. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  9. 2018Sanity Checks for Saliency Mapscited by 1,149
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  11. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  12. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
10 more
  1. 2016The Mythos of Model Interpretabilitycited by 716
  2. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  3. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  4. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  5. 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
  6. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  7. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  8. 2019CXPlain: Causal Explanations for Model Interpretation under Uncertaintycited by 98
  9. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 78
  10. 2017Interpretable & Explorable Approximations of Black Box Modelscited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 7; 2023: 16; 2024: 12; 2025: 9; 2026: 6.

59 papers.

Sort:

  1. 2021FastSHAP: Real-Time Shapley Value Estimationcited by 123
  2. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 100
  3. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  4. 2023Learning to Maximize Mutual Information for Dynamic Feature Selectioncited by 54
  5. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  6. 2022WeightedSHAP: analyzing and improving Shapley based feature attributionscited by 36
  7. 2023Explaining Predictive Uncertainty with Information Theoretic Shapley Valuescited by 36
  8. 2021Locally Sparse Neural Networks for Tabular Biomedical Datacited by 34
  9. 2024On the Challenges and Opportunities in Generative AIcited by 31
  10. 2021Rationales for Sequential Predictionscited by 26
  11. 2021What to Learn, and How: Toward Effective Learning from Rationalescited by 25
  12. 2024Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attributioncited by 21
38 more
  1. 2023CoRTX: Contrastive Framework for Real-time Explanationcited by 20
  2. 2023Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustnesscited by 20
  3. 2023ProtoGate: Prototype-based Neural Networks with Global-to-local Feature Selection for Tabular Biomedical Datacited by 18
  4. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  5. 2023Efficient XAI Techniques: A Taxonomic Surveycited by 14
  6. 2022Interpretations Steered Network Pruning via Amortized Inferred Saliency Mapscited by 11
  7. 2023Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluationcited by 11
  8. 2025Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasonscited by 11
  9. 2023Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretabilitycited by 10
  10. 2024Interpretability Needs a New Paradigmcited by 10
  11. 2023Goodhart's Law Applies to NLP's Explanation Benchmarkscited by 9
  12. 2021Attribution of Predictive Uncertainties in Classification Modelscited by 8
  13. 2024Adaptive Sampling of k-Space in Magnetic Resonance for Rapid Pathology Predictioncited by 7
  14. 2023Efficient GNN Explanation via Learning Removal-based Attributioncited by 6
  15. 2024Provably Better Explanations with Optimized Aggregation of Feature Attributionscited by 6
  16. 2024Comprehensive Attribution: Inherently Explainable Vision Model with Feature Detectorcited by 5
  17. 2021Consistent Sufficient Explanations and Minimal Local Rules for explaining regression and classification modelscited by 4
  18. 2023The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Researchcited by 4
  19. 2022New-Onset Diabetes Assessment Using Artificial Intelligence-Enhanced Electrocardiographycited by 3
  20. 2022An Additive Instance-Wise Approach to Multi-class Model Interpretationcited by 3
  21. 2023On the Feasibility of Machine Learning Augmented Magnetic Resonance for Point-of-Care Identification of Diseasecited by 3
  22. 2023A Comprehensive and Reliable Feature Attribution Method: Double-sided Remove and Reconstruct (DoRaR)cited by 3
  23. 2024Geometric Remove-and-Retrain (GOAR): Coordinate-Invariant eXplainable AI Assessmentcited by 3
  24. 2024Sufficient and Necessary Explanations (and What Lies in Between)cited by 3
  25. 2024Explanations that reveal all through the definition of encodingcited by 3
  26. 2025Prediction via Shapley Value Regressioncited by 3
  27. 2026GNN Explanations that do not Explain and How to find Themcited by 3
  28. 2021Bounded logit attention: Learning to explain image classifierscited by 2
  29. 2021When Stability meets Sufficiency: Informative Explanations that do not Overwhelmcited by 2
  30. 2023Explainable AI for clinical risk prediction: a survey of concepts, methods, and modalitiescited by 2
  31. 2024Let the Fuzzy Rule Speak: Enhancing In-context Learning Debiasing with Interpretabilitycited by 2
  32. 2025FW-Shapley: Real-time Estimation of Weighted Shapley Valuescited by 2
  33. 2025From Pixels to Perception: Interpretable Predictions via Instance-wise Grouped Feature Selectioncited by 2
  34. 2025Shift-Invariant Attribute Scoring for Kolmogorov-Arnold Networks via Shapley Valuecited by 2
  35. 2022Explainability as statistical inferencecited by 1
  36. 2024Local Feature Selection without Label or Feature Leakage for Interpretable Machine Learning Predictionscited by 1
  37. 2025Two-stage Vision Transformers and Hard Masking offer Robust Object Representationscited by 1
  38. 2026DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Datacited by 1

50 most cited of 59 All 59 (JSON)