mlsanityData to 7 Oct 2026

Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy

arXiv 2106.13200 ↗First posted 24 Jun 2021cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Deep Neural Networks (DNNs) are known to be strong predictors, but their prediction strategies can rarely be understood. With recent advances in Explainable Artificial Intelligence (XAI), approaches are available to explore the reasoning behind those complex models' predictions. Among post-hoc attribution methods, Layer-wise Relevance Propagation (LRP) shows high performance. For deeper quantitative analysis, manual approaches exist, but without the right tools they are unnecessarily labor intensive. In this software paper, we introduce three software packages targeted at scientists to explore model reasoning using attribution approaches and beyond: (1) Zennit - a highly customizable and intuitive attribution framework implementing LRP and related approaches in PyTorch, (2) CoRelAy - a framework to easily and quickly construct quantitative analysis pipelines for dataset-wide analyses of explanations, and (3) ViRelAy - a web-application to interactively explore data, attributions, and analysis results. With this, we provide a standardized implementation solution for XAI, to contribute towards more reproducibility in our field.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 44 of its 120 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

44 of 120 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  11. 2016Wide Residual Networkscited by 3,057
  12. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
32 more
  1. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  4. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  5. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  6. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  7. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  8. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  9. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  10. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 423
  11. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  12. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  13. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  14. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  15. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  16. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 248
  17. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 243
  18. 2020Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAIcited by 125
  19. 2021SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effectscited by 103
  20. 2019Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruningcited by 89
  21. 2017Learning how to explain neural networks: PatternNet and PatternAttributioncited by 84
  22. 2018How Important Is a Neuron?cited by 73
  23. 2019Towards Best Practice in Explaining Neural Network Decisions with LRPcited by 70
  24. 2018iNNvestigate neural networks!cited by 58
  25. 2015Analyzing Classifiers: Fisher Vectors and Deep Neural Networkscited by 24
  26. 2019Explaining and Interpreting LSTMscited by 23
  27. 2018Computationally Efficient Measures of Internal Neuron Importancecited by 19
  28. 2017Understanding and Comparing Deep Neural Networks for Age and Gender Classificationcited by 17
  29. 2019On the Explanation of Machine Learning Predictions in Clinical Gait Analysiscited by 15
  30. 2020Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagationcited by 11
  31. 2021Learning Domain Invariant Representations by Joint Wasserstein Distance Minimizationcited by 6
  32. 2016Controlling Explanatory Heatmap Resolution and Semantics via Decomposition Depthcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 12; 2024: 11; 2025: 15; 2026: 7.

53 papers.

Sort:

By area:

  1. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 116
  2. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  3. 2024Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarkscited by 28
  4. 2022Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspacescited by 25
  5. 2022Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergencecited by 24
  6. 2023Explainable AI for Time Series via Virtual Inspection Layerscited by 23
  7. 2022Revealing Hidden Context Bias in Segmentation and Object Detection through Concept-specific Explanationscited by 17
  8. 2023Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discoverycited by 15
  9. 2022Optimizing Explanations by Network Canonization and Hyperparameter Searchcited by 11
  10. 2024Towards Natural Machine Unlearningcited by 11
  11. 2024BEExAI: Benchmark to Evaluate Explainable AIcited by 11
  12. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
38 more
  1. 2023Preemptively Pruning Clever-Hans Strategies in Deep Neural Networkscited by 9
  2. 2024Sanity Checks Revisited: An Exploration to Repair the Model Parameter Randomisation Testcited by 9
  3. 2024A Fresh Look at Sanity Checks for Saliency Mapscited by 8
  4. 2022Explain to Not Forget: Defending Against Catastrophic Forgetting with XAIcited by 7
  5. 2022Measurably Stronger Explanation Reliability via Model Canonizationcited by 6
  6. 2023Prototypical Self-Explainable Models Without Re-trainingcited by 6
  7. 2024Sparse, Efficient and Explainable Data Attribution with DualXDAcited by 6
  8. 2024Reactive Model Correction: Mitigating Harm to Task-Relevant Features via Conditional Bias Suppressioncited by 5
  9. 2023Better Understanding Differences in Attribution Methods via Systematic Evaluationscited by 4
  10. 2023From Classification to Segmentation with Explainable AI: A Study on Crack Detection and Growth Monitoringcited by 4
  11. 2023Robustness of Visual Explanations to Common Data Augmentationcited by 3
  12. 2023XAI-based Comparison of Input Representations for Audio Event Classificationcited by 3
  13. 2025Pairwise Matching of Intermediate Representations for Fine-grained Explainabilitycited by 3
  14. 2025Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMscited by 3
  15. 2023Interpreting Deep Neural Networks with the Package innsightcited by 2
  16. 2023Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagationcited by 2
  17. 2024Locally Testing Model Detections for Semantic Global Conceptscited by 2
  18. 2026Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspacescited by 2
  19. 2023Bridging the Gap: Gaze Events as Interpretable Concepts to Explain Deep Neural Sequence Modelscited by 1
  20. 2024Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagationcited by 1
  21. 2025Post-Hoc Concept Disentanglement: From Correlated to Isolated Concept Representationscited by 1
  22. 2025PnPXAI: A Universal XAI Framework Providing Automatic Explanations Across Diverse Modalities and Modelscited by 1
  23. 2025Learning golf swing signatures from a single wrist-worn inertial sensorcited by 1
  24. 2025FaCT: Faithful Concept Traces for Explaining Neural Network Decisionscited by 1
  25. 2026From Clever Hans to Scientific Discovery: Interpreting EEG Foundational Transformers with LRPcited by 1
  26. 2026FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learningcited by 1
  27. 2022Explanation-based Counterfactual Retraining(XCR): A Calibration Method for Black-box Modelscited by 0
  28. 2023Multi-Channel Time-Series Person and Soft-Biometric Identificationcited by 0
  29. 2024Challenges in explaining deep learning models for data with biological variationcited by 0
  30. 2025Relevance-driven Input Dropout: an Explanation-guided Regularization Techniquecited by 0
  31. 2025TDHook: A Lightweight Framework for Interpretabilitycited by 0
  32. 2025LLEXICORP: End-user Explainability of Convolutional Neural Networkscited by 0
  33. 2025Beyond saliency: enhancing explanation of speech emotion recognition with expert-referenced acoustic cuescited by 0
  34. 2025Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learningcited by 0
  35. 2025Always Keep Your Promises: A Model-Agnostic Attribution Algorithm for Neural Networkscited by 0
  36. 2025Preserving Marker Specificity with Lightweight Channel-Independent Representation Learningcited by 0
  37. 2025Interpretable Plant Leaf Disease Detection Using Attention-Enhanced CNNcited by 0
  38. 2026X-SYS: A Reference Architecture for Interactive Explanation Systemscited by 0

50 most cited of 53 All 53 (JSON)