# Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification

arXiv: https://arxiv.org/abs/2011.04041 · HTML: https://mlsanity.com/paper/2011.04041/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2020.

## Built on by (most cited first, 30 of 32)

- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [Designing Inherently Interpretable Machine Learning Models](https://mlsanity.com/paper/2111.01743/) (2021)
- [Explainable, Interpretable & Trustworthy AI for Intelligent Digital Twin: Case Study on Remaining Useful Life](https://mlsanity.com/paper/2301.06676/) (2023)
- [Neural Networks are Decision Trees](https://mlsanity.com/paper/2210.05189/) (2022)
- [Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verification](https://mlsanity.com/paper/2111.08922/) (2021)
- [Do Large Language Models Know How Much They Know?](https://mlsanity.com/paper/2502.19573/) (2025)
- [GAM(e) changer or not? An evaluation of interpretable machine learning models based on additive model constraints](https://mlsanity.com/paper/2204.09123/) (2022)
- [Any Deep ReLU Network is Shallow](https://mlsanity.com/paper/2306.11827/) (2023)
- [Self-interpretable Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/2105.08589/) (2021)
- [PiML Toolbox for Interpretable Machine Learning Model Development and Diagnostics](https://mlsanity.com/paper/2305.04214/) (2023)
- [Unwrapping All ReLU Networks](https://mlsanity.com/paper/2305.09424/) (2023)
- [Interpretable Machine Learning based on Functional ANOVA Framework: Algorithms and Comparisons](https://mlsanity.com/paper/2305.15670/) (2023)
- [Using Model-Based Trees with Boosting to Fit Low-Order Functional ANOVA Models](https://mlsanity.com/paper/2207.06950/) (2022)
- [Comparing Baseline Shapley and Integrated Gradients for Local Explanation: Some Additional Insights](https://mlsanity.com/paper/2208.06096/) (2022)
- [On marginal feature attributions of tree-based models](https://mlsanity.com/paper/2302.08434/) (2023)
- [Studying Cross-cluster Modularity in Neural Networks](https://mlsanity.com/paper/2502.02470/) (2025)
- [Exact Upper and Lower Bounds for the Output Distribution of Neural Networks with Random Inputs](https://mlsanity.com/paper/2502.11672/) (2025)
- [Relating Piecewise Linear Kolmogorov Arnold Networks to ReLU Networks](https://mlsanity.com/paper/2503.01702/) (2025)
- [Can AI automatically analyze public opinion? A LLM agents-based agentic pipeline for timely public opinion analysis](https://mlsanity.com/paper/2505.11401/) (2025)
- [Interpretable (not just posthoc-explainable) medical claims modeling for discharge placement to prevent avoidable all-cause readmissions or death](https://mlsanity.com/paper/2208.12814/) (2022)
- [The Topos of Transformer Networks](https://mlsanity.com/paper/2403.18415/) (2024)
- [Explainable Bayesian deep learning through input-skip Latent Binary Bayesian Neural Networks](https://mlsanity.com/paper/2503.10496/) (2025)
- [Linear Iterative Feature Embedding: An Ensemble Framework for Interpretable Model](https://mlsanity.com/paper/2103.09983/) (2021)
- [Interpretable Feature Engineering for Time Series Predictors using Attention Networks](https://mlsanity.com/paper/2205.12723/) (2022)
- [LEURN: Learning Explainable Univariate Rules with Neural Networks](https://mlsanity.com/paper/2303.14937/) (2023)
- [The Power of Typed Affine Decision Structures: A Case Study](https://mlsanity.com/paper/2304.14888/) (2023)
- [CRITS: Convolutional Rectifier for Interpretable Time Series Classification](https://mlsanity.com/paper/2506.12042/) (2025)
- [Pathwise Explanation of ReLU Neural Networks](https://mlsanity.com/paper/2506.18037/) (2025)
- [GLAI: GreenLightningAI for Accelerated Training through Knowledge Decoupling](https://mlsanity.com/paper/2510.00883/) (2025)
- [How Understanding Forecast Uncertainty Resolves the Explainability Problem in Machine Learning Models](https://mlsanity.com/paper/2602.00179/) (2026)

## Abstract (arXiv)

The deep neural networks (DNNs) have achieved great success in learning complex patterns with strong predictive power, but they are often thought of as "black box" models without a sufficient level of transparency and interpretability. It is important to demystify the DNNs with rigorous mathematics and practical tools, especially when they are used for mission-critical applications. This paper aims to unwrap the black box of deep ReLU networks through local linear representation, which utilizes the activation pattern and disentangles the complex network into an equivalent set of local linear models (LLMs). We develop a convenient LLM-based toolkit for interpretability, diagnostics, and simplification of a pre-trained deep ReLU network. We propose the local linear profile plot and other visualization methods for interpretation and diagnostics, and an effective merging strategy for network simplification. The proposed methods are demonstrated by simulation examples, benchmark datasets, and a real case study in home lending credit risk assessment.