# A comparison of deep networks with ReLU activation function and linear spline-type methods

arXiv: https://arxiv.org/abs/1804.02253 · HTML: https://mlsanity.com/paper/1804.02253/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2018.

## Built on by (most cited first, 18 of 18)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Towards Understanding and Mitigating Audio Adversarial Examples for Speaker Recognition](https://mlsanity.com/paper/2206.03393/) (2022)
- [Precision Machine Learning](https://mlsanity.com/paper/2210.13447/) (2022)
- [TrustGuard: GNN-based Robust and Explainable Trust Evaluation with Dynamicity Support](https://mlsanity.com/paper/2306.13339/) (2023)
- [Estimation of a regression function on a manifold by fully connected deep neural networks](https://mlsanity.com/paper/2107.09532/) (2021)
- [Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks](https://mlsanity.com/paper/2305.15912/) (2023)
- [CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space](https://mlsanity.com/paper/2307.09375/) (2023)
- [Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training](https://mlsanity.com/paper/2311.18022/) (2023)
- [SEC4SR: A Security Analysis Platform for Speaker Recognition](https://mlsanity.com/paper/2109.01766/) (2021)
- [Improving Robustness Against Adversarial Attacks with Deeply Quantized Neural Networks](https://mlsanity.com/paper/2304.12829/) (2023)
- [Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local pooling](https://mlsanity.com/paper/2106.05233/) (2021)
- [Nonlocal techniques for the analysis of deep ReLU neural network approximations](https://mlsanity.com/paper/2504.04847/) (2025)
- [Does color modalities affect handwriting recognition? An empirical study on Persian handwritings using convolutional neural networks](https://mlsanity.com/paper/2307.12150/) (2023)
- [Fingerprinting Robot Movements via Acoustic Side Channel](https://mlsanity.com/paper/2209.10240/) (2022)
- [Equidistribution-based training of Free Knot Splines and ReLU Neural Networks](https://mlsanity.com/paper/2407.02153/) (2024)
- [Reconstruction of frequency-localized functions from pointwise samples via least squares and deep learning](https://mlsanity.com/paper/2502.09794/) (2025)
- [Quantitative Analysis of Deeply Quantized Tiny Neural Networks Robust to Adversarial Attacks](https://mlsanity.com/paper/2503.08973/) (2025)
- [Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions](https://mlsanity.com/paper/2605.06959/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) generate much richer function spaces than shallow networks. Since the function spaces induced by shallow networks have several approximation theoretic drawbacks, this explains, however, not necessarily the success of deep networks. In this article we take another route by comparing the expressive power of DNNs with ReLU activation function to piecewise linear spline methods. We show that MARS (multivariate adaptive regression splines) is improper learnable by DNNs in the sense that for any given function that can be expressed as a function in MARS with M parameters there exists a multilayer neural network with O(M log (M/varepsilon)) parameters that approximates this function up to sup-norm error varepsilon. We show a similar result for expansions with respect to the Faber-Schauder system. Based on this, we derive risk comparison inequalities that bound the statistical risk of fitting a neural network by the statistical risk of spline-based methods. This shows that deep networks perform better or only slightly worse than the considered spline methods. We provide a constructive proof for the function approximations.