mlsanityData to 7 Oct 2026

Interpretable Explanations of Black Boxes by Meaningful Perturbation

arXiv 1704.03296 ↗First posted 11 Apr 2017cs.CV cs.AI cs.LGcited by 611

Not on the map. Why?

Abstract (arXiv)

As machine learning algorithms are increasingly applied to high impact yet high risk tasks, such as medical diagnosis or autonomous driving, it is critical that researchers can explain how such algorithms arrived at their predictions. In recent years, a number of image saliency methods have been developed to summarize where highly complex neural networks "look" in an image for evidence for their predictions. However, these techniques are limited by their heuristic nature and architectural constraints. In this paper, we make two main contributions: First, we propose a general framework for learning different kinds of explanations for any black box algorithm. Second, we specialise the framework to find the part of an image most responsible for a classifier decision. Unlike previous works, our method is model-agnostic and testable because it is grounded in explicit and interpretable image perturbations.

Key facts. Cited by 611 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 131; 2022: 121; 2023: 109; 2024: 107; 2025: 76; 2026: 66.

611 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  3. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  4. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  5. 2021Explainable artificial intelligence (XAI) in deep learning-based medical image analysiscited by 147
  6. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  7. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  8. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 116
  9. 2022Algorithms to estimate Shapley value feature attributionscited by 114
  10. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  11. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  12. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
38 more
  1. 2021BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentationcited by 104
  2. 2021Guided Integrated Gradients: An Adaptive Path Method for Removing Noisecited by 102
  3. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 100
  4. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  5. 2021Interpretable Image Classification with Differentiable Prototypes Assignmentcited by 97
  6. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
  7. 2024The Hidden Attention of Mamba Modelscited by 88
  8. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  9. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  10. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  11. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  12. 2022Impossibility Theorems for Feature Attributioncited by 78
  13. 2021Explaining Time Series Predictions with Dynamic Maskscited by 72
  14. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  15. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  16. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 67
  17. 2021Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysiscited by 65
  18. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  19. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  20. 2022Concept Activation Regions: A Generalized Framework For Concept-Based Explanationscited by 57
  21. 2021Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanationscited by 56
  22. 2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 49
  23. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  24. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  25. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  26. 2021Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directionscited by 41
  27. 2022Contrastive language and vision learning of general fashion conceptscited by 41
  28. 2021STEEX: Steering Counterfactual Explanations with Semanticscited by 40
  29. 2021Biometrics: Trust, but Verifycited by 39
  30. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 38
  31. 2022Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysiscited by 38
  32. 2023Explain Any Concept: Segment Anything Meets Concept-Based Explanationcited by 37
  33. 2021Defining and Quantifying the Emergence of Sparse Concepts in DNNscited by 36
  34. 2022Missingness Bias in Model Debuggingcited by 36
  35. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  36. 2022Explaining Deepfake Detection by Analysing Image Matchingcited by 34
  37. 2024Gradient based Feature Attribution in Explainable AI: A Technical Reviewcited by 34
  38. 2022Making Sense of Dependence: Efficient Black-box Explanations Using Dependence Measurecited by 33

50 most cited of 611 All 611 (JSON)