# Any Deep ReLU Network is Shallow

arXiv: https://arxiv.org/abs/2306.11827 · HTML: https://mlsanity.com/paper/2306.11827/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 9 of 9)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification](https://mlsanity.com/paper/2011.04041/) (2020)
- [Improved Bounds on Neural Complexity for Representing Piecewise Linear Functions](https://mlsanity.com/paper/2210.07236/) (2022)
- [Empirical Bounds on Linear Regions of Deep Rectifier Networks](https://mlsanity.com/paper/1810.03370/) (2018)
- [ReLU Deep Neural Networks from the Hierarchical Basis Perspective](https://mlsanity.com/paper/2105.04156/) (2021)
- [Unwrapping All ReLU Networks](https://mlsanity.com/paper/2305.09424/) (2023)
- [How to Explain Neural Networks: an Approximation Perspective](https://mlsanity.com/paper/2105.07831/) (2021)

## Built on by (most cited first, 6 of 6)

- [The structure of the token space for large language models](https://mlsanity.com/paper/2410.08993/) (2024)
- [Relating Piecewise Linear Kolmogorov Arnold Networks to ReLU Networks](https://mlsanity.com/paper/2503.01702/) (2025)
- [The Topos of Transformer Networks](https://mlsanity.com/paper/2403.18415/) (2024)
- [Certified Neural Approximations of Nonlinear Dynamics](https://mlsanity.com/paper/2505.15497/) (2025)
- [Parallel Algorithms for Exact Enumeration of Deep Neural Network Activation Regions](https://mlsanity.com/paper/2403.00860/) (2024)
- [RepAct: The Re-parameterizable Adaptive Activation Function](https://mlsanity.com/paper/2407.00131/) (2024)

## Abstract (arXiv)

We constructively prove that every deep ReLU network can be rewritten as a functionally identical three-layer network with weights valued in the extended reals. Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. The resulting shallow network is transparent and used to generate explanations of the model s behaviour.