# Explaining Neural Networks without Access to Training Data

arXiv: https://arxiv.org/abs/2206.04891 · HTML: https://mlsanity.com/paper/2206.04891/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Zero-Shot Knowledge Distillation in Deep Networks](https://mlsanity.com/paper/1905.08114/) (2019)
- [Interpreting CNNs via Decision Trees](https://mlsanity.com/paper/1802.00121/) (2018)
- [Improving the Interpretability of Deep Neural Networks with Knowledge Distillation](https://mlsanity.com/paper/1812.10924/) (2018)
- [Dream Distillation: A Data-Independent Model Compression Framework](https://mlsanity.com/paper/1905.07072/) (2019)
- [Towards Interpretable ANNs: An Exact Transformation to Multi-Class Multivariate Decision Trees](https://mlsanity.com/paper/2003.04675/) (2020)

## Built on by (most cited first, 1 of 1)

- [Privacy Risks and Preservation Methods in Explainable Artificial Intelligence: A Scoping Review](https://mlsanity.com/paper/2505.02828/) (2025)

## Abstract (arXiv)

We consider generating explanations for neural networks in cases where the network's training data is not accessible, for instance due to privacy or safety issues. Recently, I-Nets have been proposed as a sample-free approach to post-hoc, global model interpretability that does not require access to training data. They formulate interpretation as a machine learning task that maps network representations (parameters) to a representation of an interpretable function. In this paper, we extend the I-Net framework to the cases of standard and soft decision trees as surrogate models. We propose a suitable decision tree representation and design of the corresponding I-Net output layers. Furthermore, we make I-Nets applicable to real-world tasks by considering more realistic distributions when generating the I-Net's training data. We empirically evaluate our approach against traditional global, post-hoc interpretability approaches and show that it achieves superior results when the training data is not accessible.