# SkelEx and BoundEx: Natural Visualization of ReLU Neural Networks

arXiv: https://arxiv.org/abs/2305.05562 · HTML: https://mlsanity.com/paper/2305.05562/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 12 of 12)

- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [Complexity of Linear Regions in Deep Networks](https://mlsanity.com/paper/1901.09021/) (2019)
- [Bounding and Counting Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1711.02114/) (2017)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [Empirical Studies on the Properties of Linear Regions in Deep Neural Networks](https://mlsanity.com/paper/2001.01072/) (2020)
- [Hold me tight! Influence of discriminative features on deep network boundaries](https://mlsanity.com/paper/2002.06349/) (2020)
- [Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary](https://mlsanity.com/paper/2009.07974/) (2020)
- [Are All Linear Regions Created Equal?](https://mlsanity.com/paper/2202.11749/) (2022)
- [Optimal Function Approximation with Relu Neural Networks](https://mlsanity.com/paper/1909.03731/) (2019)
- [Super-k: A Piecewise Linear Classifier Based on Voronoi Tessellations](https://mlsanity.com/paper/2012.15492/) (2020)

## Abstract (arXiv)

Despite their limited interpretability, weights and biases are still the most popular encoding of the functions learned by ReLU Neural Networks (ReLU NNs). That is why we introduce SkelEx, an algorithm to extract a skeleton of the membership functions learned by ReLU NNs, making those functions easier to interpret and analyze. To the best of our knowledge, this is the first work that considers linear regions from the perspective of critical points. As a natural follow-up, we also introduce BoundEx, which is the first analytical method known to us to extract the decision boundary from the realization of a ReLU NN. Both of those methods introduce very natural visualization tool for ReLU NNs trained on low-dimensional data.