# Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression

arXiv: https://arxiv.org/abs/2609.25710 · HTML: https://mlsanity.com/paper/2609.25710/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 50 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 5 of 5)

- [Covering Numbers for Deep ReLU Networks with Applications to Function Approximation and Nonparametric Regression](https://mlsanity.com/paper/2410.06378/) (2024)
- [Optimal Neural Network Approximation for High-Dimensional Continuous Functions](https://mlsanity.com/paper/2409.02363/) (2024)
- [Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations](https://mlsanity.com/paper/2602.19691/) (2026)
- [Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy](https://mlsanity.com/paper/2606.16975/) (2026)
- [On Explicit Super-Expressive Approximation for Neural Networks](https://mlsanity.com/paper/2607.06781/) (2026)

## Abstract (arXiv)

The statistical accuracy of neural networks depends on both their approximation power and the complexity of the class fitted from data. While increasing network size is a natural way to improve approximation, parameter magnitude provides another resource whose role must be quantified in both respects. We establish a sharp width--magnitude tradeoff at fixed depth using one elementary bounded 1-Lipschitz Dyadic--Triangular Activation. For the unit β-H"older ball on [0,1]d with 0<β≤1, the optimal Lp approximation error for 0<p<∞ is of order [N2log(eNT)]-β/d when the network width satisfies N≥2d+3 and the parameter magnitudes are bounded by T≥1. Matching lower bounds hold for every fixed globally H"older activation; its H"older exponent affects the constants but not the rate. Under bounded design densities and independent centered sub-Gaussian noise, approximate least squares over the full clipped class at depth 23 attains the classical H"older minimax risk O(M^-frac2β2β+d) without logarithmic loss whenever N2log(eNT)asymp M^fracd2β+d, where M is the sample size. This yields a continuum of statistically optimal choices, ranging from unit parameter radius to fixed network size. At fixed size, four hidden layers with at most 8d+7 nonzero parameters give a near-optimal radius, while six layers with at most 8d+27 attain the optimal order log T=O(η-d/β) at approximation error η. The same decoding method also yields fixed-size Transformer approximation.