mlsanityData to 7 Oct 2026

The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

arXiv 2105.14944 ↗First posted 31 May 2021cs.CV cs.AI cs.HCcited by 87

Not on the map. Why?

Abstract (arXiv)

Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications. Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work. However, despite humans being the target end-users, most attribution methods were only evaluated on proxy automatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et al. 2018). In this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine-grained classification, and when an image is natural or adversarial (i.e., contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-AI teams compared to AI alone. Importantly, we found automatic attribution-map evaluation measures to correlate poorly with the actual human-AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-loop applications and to rethink the existing evaluation metrics.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 58 of its 84 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

58 of 84 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  12. 2018Sanity Checks for Saliency Mapscited by 1,149
38 more
  1. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  2. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  3. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  4. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  5. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  6. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  7. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  8. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
  9. 2015Towards Open Set Deep Networkscited by 533
  10. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 459
  11. 2017Interpretation of Neural Networks is Fragilecited by 443
  12. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  13. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  14. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 343
  15. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  16. 2019Counterfactual Visual Explanationscited by 312
  17. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  18. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  19. 2020Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Surveycited by 252
  20. 2019A Simple Pooling-Based Design for Real-Time Salient Object Detectioncited by 211
  21. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  22. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  23. 2016Generating Visual Explanationscited by 156
  24. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  25. 2018Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examplescited by 98
  26. 2019Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screeningcited by 94
  27. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  28. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  29. 2019Understanding Neural Networks via Feature Visualization: A surveycited by 65
  30. 2020There and Back Again: Revisiting Backpropagation Saliency Methodscited by 60
  31. 2020Towards falsifiable interpretability researchcited by 53
  32. 2018What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Playcited by 52
  33. 2018Do Explanations make VQA Models more Predictable to a Human?cited by 51
  34. 2020This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognitioncited by 50
  35. 2019Quantifying Interpretability and Trust in Machine Learning Systemscited by 49
  36. 2020How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labelscited by 39
  37. 2018Generating Counterfactual Explanations with Natural Languagecited by 37
  38. 2020Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Predictioncited by 36

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 24; 2024: 16; 2025: 16; 2026: 13.

87 papers.

Sort:

By area:

  1. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  2. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  3. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  4. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  5. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  6. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 67
  7. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  8. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  9. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  10. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  11. 2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 49
  12. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
38 more
  1. 2022Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysiscited by 38
  2. 2022Visual correspondence-based explanations improve AI robustness and human-AI team accuracycited by 37
  3. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  4. 2021Explaining Latent Representations with a Corpus of Examplescited by 30
  5. 2022A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Makingcited by 29
  6. 2022OCTET: Object-aware Counterfactual Explanationscited by 28
  7. 2023Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimizationcited by 26
  8. 2023Red Teaming Deep Neural Networks with Feature Synthesis Toolscited by 21
  9. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  10. 2022Machine Explanations and Human Understandingcited by 20
  11. 2024Interpretable LLM-based Table Question Answeringcited by 18
  12. 2022On the Effect of Information Asymmetry in Human-AI Teamscited by 17
  13. 2023The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifierscited by 15
  14. 2024Understanding Visual Feature Reliance through the Lens of Complexitycited by 15
  15. 2023Humans, AI, and Context: Understanding End-Users' Trust in a Real-World Computer Vision Applicationcited by 14
  16. 2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
  17. 2025Interactive Medical Image Analysis with Concept-based Similarity Reasoningcited by 11
  18. 2021When and How to Fool Explainable Models (and Humans) with Adversarial Examplescited by 10
  19. 2023Keep the Faith: Faithful Explanations in Convolutional Neural Networks for Case-Based Reasoningcited by 10
  20. 2022How explainable are adversarially-robust CNNs?cited by 9
  21. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  22. 2025HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputscited by 9
  23. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 9
  24. 2023Interpretability-Aware Vision Transformercited by 8
  25. 2025FACE: Faithful Automatic Concept Extractioncited by 8
  26. 2023Explaining RL Decisions with Trajectoriescited by 7
  27. 2025Towards Desiderata-Driven Design of Visual Counterfactual Explainerscited by 7
  28. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  29. 2024How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency mapscited by 6
  30. 2025B-score: Detecting biases in large language models using response historycited by 6
  31. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attributioncited by 6
  32. 2023May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainabilitycited by 5
  33. 2024Feature Accentuation: Revealing 'What' Features Respond to in Natural Imagescited by 5
  34. 2024Allowing humans to interactively guide machines where to look does not always improve human-AI team's classification accuracycited by 5
  35. 2024SLIM: Spuriousness Mitigation with Minimal Human Annotationscited by 5
  36. 2024On the Evaluation Consistency of Attribution-based Explanationscited by 5
  37. 2023Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goalscited by 4
  38. 2023Learning Human-Compatible Representations for Case-Based Decision Supportcited by 4

50 most cited of 87 All 87 (JSON)