mlsanityData to 14 Oct 2026

HIVE: Evaluating the Human Interpretability of Visual Explanations

arXiv 2112.03184 ↗First posted 6 Dec 2021cs.CVcited by 105

Not on the map. Why?

Abstract (arXiv)

As AI technology is increasingly applied to high-impact, high-risk domains, there have been a number of new methods aimed at making AI models more human interpretable. Despite the recent growth of interpretability work, there is a lack of systematic evaluation of proposed techniques. In this work, we introduce HIVE (Human Interpretability of Visual Explanations), a novel human evaluation framework that assesses the utility of explanations to human users in AI-assisted decision making scenarios, and enables falsifiable hypothesis testing, cross-method comparison, and human-centered evaluation of visual interpretability methods. To the best of our knowledge, this is the first work of its kind. Using HIVE, we conduct IRB-approved human studies with nearly 1000 participants and evaluate four methods that represent the diversity of computer vision interpretability works: GradCAM, BagNet, ProtoPNet, and ProtoTree. Our results suggest that explanations engender human trust, even for incorrect predictions, yet are not distinct enough for users to distinguish between correct and incorrect predictions. We open-source HIVE to enable future studies and encourage more human-centered approaches to interpretability research.

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). Builds on 55 of its 88 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

55 of 88 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  8. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  9. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  10. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  11. 2018Sanity Checks for Saliency Mapscited by 1,156
  12. 2020Concept Bottleneck Modelscited by 1,032
38 more
  1. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  2. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  3. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  4. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  5. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  6. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  7. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  8. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
  9. 2016Top-down Neural Attention by Excitation Backpropcited by 321
  10. 2019Counterfactual Visual Explanationscited by 313
  11. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  12. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  13. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 259
  14. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  15. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  16. 2020Neural Prototype Trees for Interpretable Fine-grained Image Recognitioncited by 214
  17. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  18. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  19. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  20. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 165
  21. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  22. 2019Seeing What a GAN Cannot Generatecited by 148
  23. 2020Debugging Tests for Model Explanationscited by 147
  24. 2018Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networkscited by 129
  25. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  26. 2021Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypescited by 120
  27. 2021Do Concept Bottleneck Models Learn as Intended?cited by 116
  28. 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
  29. 2020Estimating Example Difficulty Using Variance of Gradientscited by 97
  30. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  31. 2019Benchmarking Attribution Methods with Relative Feature Importancecited by 89
  32. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 88
  33. 2021This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networkscited by 65
  34. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  35. 2021Convolutional Dynamic Alignment Networks for Interpretable Classificationscited by 54
  36. 2020Towards falsifiable interpretability researchcited by 53
  37. 2020SCOUT: Self-aware Discriminant Counterfactual Explanationscited by 52
  38. 2020How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labelscited by 39

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 28; 2024: 33; 2025: 24; 2026: 9.

105 papers.

Sort:

By area:

  1. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 178
  2. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 109
  3. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 85
  4. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  5. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
  6. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  7. 2022ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognitioncited by 63
  8. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  9. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  10. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  11. 2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 50
  12. 2023Adversarial Counterfactual Visual Explanationscited by 47
38 more
  1. 2022Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capabilitycited by 46
  2. 2022Visual correspondence-based explanations improve AI robustness and human-AI team accuracycited by 39
  3. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  4. 2022Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysiscited by 38
  5. 2022Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networkscited by 37
  6. 2023This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizationscited by 36
  7. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32
  8. 2023Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimizationcited by 27
  9. 2023ICICLE: Interpretable Class Incremental Continual Learningcited by 24
  10. 2024LLM-Generated Black-box Explanations Can Be Adversarially Helpfulcited by 21
  11. 2022Machine Explanations and Human Understandingcited by 20
  12. 2023Scale Alone Does not Improve Mechanistic Interpretability in Vision Modelscited by 18
  13. 2023Pixel-Grounded Prototypical Part Networkscited by 18
  14. 2024Interpretable LLM-based Table Question Answeringcited by 18
  15. 2022Concept-based Explanations for Out-Of-Distribution Detectorscited by 16
  16. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  17. 2023The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifierscited by 15
  18. 2024LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Visioncited by 15
  19. 2024Understanding Visual Feature Reliance through the Lens of Complexitycited by 15
  20. 2023Humans, AI, and Context: Understanding End-Users' Trust in a Real-World Computer Vision Applicationcited by 14
  21. 2025Show and Tell: Visually Explainable Deep Neural Nets via Spatially-Aware Concept Bottleneck Modelscited by 12
  22. 2023On the Interpretability of Part-Prototype Based Classifiers: A Human Centric Analysiscited by 11
  23. 2023Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable and Trustworthy Image Recognitioncited by 11
  24. 2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
  25. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 10
  26. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  27. 2024Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspectivecited by 9
  28. 2025HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputscited by 9
  29. 2024Beyond Pixels: Enhancing LIME with Hierarchical Features and Segmentation Foundation Modelscited by 8
  30. 2024ProtoS-ViT: Visual foundation models for sparse self-explainable classificationscited by 8
  31. 2024InfoDisent: Explainability of Image Classification Models by Information Disentanglementcited by 8
  32. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attributioncited by 8
  33. 2023Take 5: Interpretable Image Classification with a Handful of Featurescited by 7
  34. 2024Post-hoc Part-prototype Networkscited by 7
  35. 2023Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groupscited by 6
  36. 2024How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency mapscited by 6
  37. 2024The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systemscited by 6
  38. 2024Feature Accentuation: Revealing 'What' Features Respond to in Natural Imagescited by 5

50 most cited of 105 All 105 (JSON)