# Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution

arXiv: https://arxiv.org/abs/1802.06259 · HTML: https://mlsanity.com/paper/1802.06259/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2018.

## Built on by (most cited first, 30 of 32)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Trustworthy Graph Neural Networks: Aspects, Methods and Trends](https://mlsanity.com/paper/2205.07424/) (2022)
- [Robust Counterfactual Explanations on Graph Neural Networks](https://mlsanity.com/paper/2107.04086/) (2021)
- [A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions](https://mlsanity.com/paper/2202.11912/) (2022)
- [Scalable Rule-Based Representation Learning for Interpretable Classification](https://mlsanity.com/paper/2109.15103/) (2021)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [Do Sparse Autoencoders Capture Concept Manifolds?](https://mlsanity.com/paper/2604.28119/) (2026)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Learning Interpretable Rules for Scalable Data Representation and Classification](https://mlsanity.com/paper/2310.14336/) (2023)
- [Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verification](https://mlsanity.com/paper/2111.08922/) (2021)
- [Deep Contract Design via Discontinuous Networks](https://mlsanity.com/paper/2307.02318/) (2023)
- [Non-Linear Coordination Graphs](https://mlsanity.com/paper/2211.08404/) (2022)
- [Learning and Meshing from Deep Implicit Surface Networks Using an Efficient Implementation of Analytic Marching](https://mlsanity.com/paper/2106.10031/) (2021)
- [A Survey of Neural Trees](https://mlsanity.com/paper/2209.03415/) (2022)
- [Deep Active Learning for Text Classification with Diverse Interpretations](https://mlsanity.com/paper/2108.10687/) (2021)
- [Deep ReLU Networks Have Surprisingly Simple Polytopes](https://mlsanity.com/paper/2305.09145/) (2023)
- [Generating Post-hoc Explanations for Skip-gram-based Node Embeddings by Identifying Important Nodes with Bridgeness](https://mlsanity.com/paper/2304.12036/) (2023)
- [Finding Representative Interpretations on Convolutional Neural Networks](https://mlsanity.com/paper/2108.06384/) (2021)
- [Regime Change Hypothesis: Foundations for Decoupled Dynamics in Neural Network Training](https://mlsanity.com/paper/2602.08333/) (2026)
- [Manifoldron: Direct Space Partition via Manifold Discovery](https://mlsanity.com/paper/2201.05279/) (2022)
- [How to Explain Neural Networks: an Approximation Perspective](https://mlsanity.com/paper/2105.07831/) (2021)
- [On the Bias-Variance Characteristics of LIME and SHAP in High Sparsity Movie Recommendation Explanation Tasks](https://mlsanity.com/paper/2206.04784/) (2022)
- [Diffusion Integrated Gradients: Controllable Path Generation for Flexible Feature Attribution](https://mlsanity.com/paper/2606.22314/) (2026)
- [DNN2LR: Automatic Feature Crossing for Credit Scoring](https://mlsanity.com/paper/2102.12036/) (2021)
- [Training Neural Networks for Solving 1-D Optimal Piecewise Linear Approximation](https://mlsanity.com/paper/2110.08259/) (2021)
- [Model Doctor for Diagnosing and Treating Segmentation Error](https://mlsanity.com/paper/2302.08980/) (2023)
- [The Power of Typed Affine Decision Structures: A Case Study](https://mlsanity.com/paper/2304.14888/) (2023)
- [A max-affine spline approximation of neural networks using the Legendre transform of a convex-concave representation](https://mlsanity.com/paper/2307.09602/) (2023)
- [Interpretable Classification via a Rule Network with Selective Logical Operators](https://mlsanity.com/paper/2408.11918/) (2024)
- [ReLU Networks as Random Functions: Their Distribution in Probability Space](https://mlsanity.com/paper/2503.22082/) (2025)

## Abstract (arXiv)

Strong intelligent machines powered by deep neural networks are increasingly deployed as black boxes to make decisions in risk-sensitive domains, such as finance and medical. To reduce potential risk and build trust with users, it is critical to interpret how such machines make their decisions. Existing works interpret a pre-trained neural network by analyzing hidden neurons, mimicking pre-trained models or approximating local predictions. However, these methods do not provide a guarantee on the exactness and consistency of their interpretation. In this paper, we propose an elegant closed form solution named OpenBox to compute exact and consistent interpretations for the family of Piecewise Linear Neural Networks (PLNN). The major idea is to first transform a PLNN into a mathematically equivalent set of linear classifiers, then interpret each linear classifier by the features that dominate its prediction. We further apply OpenBox to demonstrate the effectiveness of non-negative and sparse constraints on improving the interpretability of PLNNs. The extensive experiments on both synthetic and real world data sets clearly demonstrate the exactness and consistency of our interpretation.