mlsanityData to 7 Oct 2026

Benchmarking and Survey of Explanation Methods for Black Box Models

arXiv 2102.13076 ↗First posted 25 Feb 2021cs.AI cs.CY cs.LGcited by 136

Not on the map. Why?

Abstract (arXiv)

The widespread adoption of black-box models in Artificial Intelligence has enhanced the need for explanation methods to reveal how these obscure models reach specific decisions. Retrieving explanations is fundamental to unveil possible biases and to resolve practical or ethical issues. Nowadays, the literature is full of methods with different explanations. We provide a categorization of explanation methods based on the type of explanation returned. We present the most recent and widely used explainers, and we show a visual comparison among explanations and a quantitative benchmarking.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Builds on 93 of its 196 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

93 of 196 references.

References · 93

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  9. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  12. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
38 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  4. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  5. 2018Sanity Checks for Saliency Mapscited by 1,149
  6. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  7. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  8. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  9. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  10. 2019Attention is not Explanationcited by 916
  11. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  12. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  13. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  14. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 612
  15. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
  16. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  17. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  18. 2016Tutorial on Variational Autoencoderscited by 404
  19. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  20. 2020Explainability in Graph Neural Networks: A Taxonomic Surveycited by 383
  21. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  22. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  23. 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 318
  24. 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
  25. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  26. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  27. 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
  28. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 243
  29. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  30. 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
  31. 2019FACE: Feasible and Actionable Counterfactual Explanationscited by 232
  32. 2016Understanding Neural Networks through Representation Erasurecited by 223
  33. 2019Interpretable Counterfactual Explanations Guided by Prototypescited by 223
  34. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  35. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 213
  36. 2016Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Modelscited by 212
  37. 2020XGNN: Towards Model-Level Explanations of Graph Neural Networkscited by 211
  38. 2018Explainable Recommendation: A Survey and New Perspectivescited by 210

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 15; 2023: 27; 2024: 37; 2025: 28; 2026: 18.

136 papers.

Sort:

  1. 2021Trustworthy AI: From Principles to Practicescited by 206
  2. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  3. 2023Concept-based Explainable Artificial Intelligence: A Surveycited by 97
  4. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
  5. 2023Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistencycited by 42
  6. 2021ECINN: Efficient Counterfactuals from Invertible Neural Networkscited by 26
  7. 2023Opti-CAM: Optimizing saliency maps for interpretabilitycited by 25
  8. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  9. 2021Counterfactual Explanations for Machine Learning: Challenges Revisitedcited by 19
  10. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  11. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  12. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
38 more
  1. 2022Deletion and Insertion Tests in Regression Modelscited by 14
  2. 2021Thermostat: A Large Collection of NLP Model Explanations and Analysis Toolscited by 11
  3. 2023Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainabilitycited by 11
  4. 2024Attention Meets Post-hoc Interpretability: A Mathematical Perspectivecited by 11
  5. 2021Conclusive Local Interpretation Rules for Random Forestscited by 9
  6. 2022s-LIME: Reconciling Locality and Fidelity in Linear Explanationscited by 8
  7. 2023Sanity checks for patch visualisation in prototype-based image classificationcited by 8
  8. 2024Exploiting Preference Elicitation in Interactive and User-centered Algorithmic Recourse: An Initial Explorationcited by 8
  9. 2024Interpretable Clustering: A Surveycited by 8
  10. 2022Benchmarking Instance-Centric Counterfactual Algorithms for XAI: From White Box to Black Boxcited by 7
  11. 2022SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usabilitycited by 7
  12. 2023Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spacescited by 7
  13. 2024A multi-criteria approach for selecting an explanation from the set of counterfactuals produced by an ensemble of explainerscited by 7
  14. 2024XEdgeAI: A Human-centered Industrial Inspection Framework with Data-centric Explainable Edge AI Approachcited by 7
  15. 2024Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Featurescited by 7
  16. 2022GRETEL: A unified framework for Graph Counterfactual Explanation Evaluationcited by 6
  17. 2023Human Attention-Guided Explainable Artificial Intelligence for Computer Vision Modelscited by 6
  18. 2023Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editorscited by 6
  19. 2023Prototypical Self-Explainable Models Without Re-trainingcited by 6
  20. 2024Distributional Counterfactual Explanations With Optimal Transportcited by 6
  21. 2024Generating Likely Counterfactuals Using Sum-Product Networkscited by 6
  22. 2024A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasurescited by 6
  23. 2024GLANCE: Global Actions in a Nutshell for Counterfactual Explainabilitycited by 6
  24. 2021A Review of Explainable Artificial Intelligence in Manufacturingcited by 5
  25. 2021XPROAX-Local explanations for text classification with progressive neighborhood approximationcited by 5
  26. 2022Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraintcited by 5
  27. 2023May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainabilitycited by 5
  28. 2025Mathematical Foundation of Interpretable Equivariant Surrogate Modelscited by 5
  29. 2025The Procedural Content Generation Benchmark: An Open-source Testbed for Generative Challenges in Gamescited by 5
  30. 2025MASCOTS: Model-Agnostic Symbolic COunterfactual explanations for Time Seriescited by 5
  31. 2023Glocal Explanations of Expected Goal Models in Soccercited by 4
  32. 2023The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Researchcited by 4
  33. 2024Path Choice Matters for Clear Attribution in Path Methodscited by 4
  34. 2024Even-if Explanations: Formal Foundations, Priorities and Complexitycited by 4
  35. 2021Attention-like feature explanation for tabular datacited by 3
  36. 2022An Additive Instance-Wise Approach to Multi-class Model Interpretationcited by 3
  37. 2023Sanity checks and improvements for patch visualisation in prototype-based image classificationcited by 3
  38. 2023Designing a Direct Feedback Loop between Humans and Convolutional Neural Networks through Local Explanationscited by 3

50 most cited of 136 All 136 (JSON)