mlsanityData to 14 Oct 2026

Any Deep ReLU Network is Shallow

arXiv 2306.11827 ↗First posted 20 Jun 2023cs.LG cs.AI stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

We constructively prove that every deep ReLU network can be rewritten as a functionally identical three-layer network with weights valued in the extended reals. Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. The resulting shallow network is transparent and used to generate explanations of the model s behaviour.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 9 of its 13 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

9 of 13 references.

References · 9

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  3. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  4. 2020Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplificationcited by 32
  5. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  6. 2018Empirical Bounds on Linear Regions of Deep Rectifier Networkscited by 23
  7. 2021ReLU Deep Neural Networks from the Hierarchical Basis Perspectivecited by 12
  8. 2023Unwrapping All ReLU Networkscited by 4
  9. 2021How to Explain Neural Networks: an Approximation Perspectivecited by 1

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 2.

6 papers.

  1. 2024The structure of the token space for large language modelscited by 10
  2. 2025Relating Piecewise Linear Kolmogorov Arnold Networks to ReLU Networkscited by 2
  3. 2024The Topos of Transformer Networkscited by 1
  4. 2025Certified Neural Approximations of Nonlinear Dynamicscited by 1
  5. 2024Parallel Algorithms for Exact Enumeration of Deep Neural Network Activation Regionscited by 0
  6. 2024RepAct: The Re-parameterizable Adaptive Activation Functioncited by 0