# ONNXExplainer: an ONNX Based Generic Framework to Explain Neural Networks Using Shapley Values

arXiv: https://arxiv.org/abs/2309.16916 · HTML: https://mlsanity.com/paper/2309.16916/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers](https://mlsanity.com/paper/1604.00825/) (2016)
- [One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques](https://mlsanity.com/paper/1909.03012/) (2019)
- [When Explanations Lie: Why Many Modified BP Attributions Fail](https://mlsanity.com/paper/1912.09818/) (2019)
- [How Important Is a Neuron?](https://mlsanity.com/paper/1805.12233/) (2018)
- [Contrastive Explanation: A Structural-Model Approach](https://mlsanity.com/paper/1811.03163/) (2018)
- [dalex: Responsible Machine Learning with Interactive Explainability and Fairness in Python](https://mlsanity.com/paper/2012.14406/) (2020)
- [DALEX: explainers for complex predictive models](https://mlsanity.com/paper/1806.08915/) (2018)
- [OmniXAI: A Library for Explainable AI](https://mlsanity.com/paper/2206.01612/) (2022)
- [Global Model Interpretation via Recursive Partitioning](https://mlsanity.com/paper/1802.04253/) (2018)
- [A Comparison of Decision Forest Inference Platforms from A Database Perspective](https://mlsanity.com/paper/2302.04430/) (2023)

## Abstract (arXiv)

Understanding why a neural network model makes certain decisions can be as important as the inference performance. Various methods have been proposed to help practitioners explain the prediction of a neural network model, of which Shapley values are most popular. SHAP package is a leading implementation of Shapley values to explain neural networks implemented in TensorFlow or PyTorch but lacks cross-platform support, one-shot deployment and is highly inefficient. To address these problems, we present the ONNXExplainer, which is a generic framework to explain neural networks using Shapley values in the ONNX ecosystem. In ONNXExplainer, we develop its own automatic differentiation and optimization approach, which not only enables One-Shot Deployment of neural networks inference and explanations, but also significantly improves the efficiency to compute explanation with less memory consumption. For fair comparison purposes, we also implement the same optimization in TensorFlow and PyTorch and measure its performance against the current state of the art open-source counterpart, SHAP. Extensive benchmarks demonstrate that the proposed optimization approach improves the explanation latency of VGG19, ResNet50, DenseNet201, and EfficientNetB0 by as much as 500%.