# On the expressivity of deep Heaviside networks

arXiv: https://arxiv.org/abs/2505.00110 · HTML: https://mlsanity.com/paper/2505.00110/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2025.

## Built on by (most cited first, 3 of 3)

- [Deep Network Trainability via Persistent Subspace Orthogonality](https://mlsanity.com/paper/2508.02882/) (2025)
- [A Composite Activation Function for Learning Stable Binary Representations](https://mlsanity.com/paper/2605.11558/) (2026)
- [Learning piecewise-smooth dynamical systems](https://mlsanity.com/paper/2608.19785/) (2026)

## Abstract (arXiv)

We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model.