# Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms

arXiv: https://arxiv.org/abs/2009.06132 · HTML: https://mlsanity.com/paper/2009.06132/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [Characterization of the Variation Spaces Corresponding to Shallow Neural Networks](https://mlsanity.com/paper/2106.15002/) (2021)
- [Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape](https://mlsanity.com/paper/2402.01258/) (2024)
- [Penalising the biases in norm regularisation enforces sparsity](https://mlsanity.com/paper/2303.01353/) (2023)
- [Generalization Error Bounds for Deep Neural Networks Trained by SGD](https://mlsanity.com/paper/2206.03299/) (2022)
- [Approximation of Functionals by Neural Network without Curse of Dimensionality](https://mlsanity.com/paper/2205.14421/) (2022)
- [Approximation results for Gradient Descent trained Shallow Neural Networks in 1d](https://mlsanity.com/paper/2209.08399/) (2022)
- [Space-Time Approximation with Shallow Neural Networks in Fourier Lebesgue spaces](https://mlsanity.com/paper/2312.08461/) (2023)
- [Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality](https://mlsanity.com/paper/2209.01173/) (2022)
- [Embeddings between Barron spaces with higher order activation functions](https://mlsanity.com/paper/2305.15839/) (2023)
- [Minimum norm interpolation by perceptra: Explicit regularization and implicit bias](https://mlsanity.com/paper/2311.06138/) (2023)
- [Delay-SDE-net: A deep learning approach for time series modelling with memory and uncertainty estimates](https://mlsanity.com/paper/2303.08587/) (2023)
- [Embedding Inequalities for Barron-type Spaces](https://mlsanity.com/paper/2305.19082/) (2023)
- [Approximation Results for Gradient Descent trained Neural Networks](https://mlsanity.com/paper/2309.04860/) (2023)
- [Solving the Poisson Equation with Dirichlet data by shallow ReLUα-networks: A regularity and approximation perspective](https://mlsanity.com/paper/2412.07728/) (2024)
- [Approximation and Gradient Descent Training with Neural Networks](https://mlsanity.com/paper/2405.11696/) (2024)
- [Curse of Dimensionality in Neural Network Optimization](https://mlsanity.com/paper/2502.05360/) (2025)
- [Statistical Learning Guarantees for Group-Invariant Barron Functions](https://mlsanity.com/paper/2509.23474/) (2025)
- [Elliptic Regularity Theory in Barron Spaces and Applications to the Deep Ritz Method](https://mlsanity.com/paper/2607.25100/) (2026)
- [Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networks](https://mlsanity.com/paper/2604.00505/) (2026)

## Abstract (arXiv)

A simple approach is proposed to obtain complexity controls for neural networks with general activation functions. The approach is motivated by approximating the general activation functions with one-dimensional ReLU networks, which reduces the problem to the complexity controls of ReLU networks. Specifically, we consider two-layer networks and deep residual networks, for which path-based norms are derived to control complexities. We also provide preliminary analyses of the function spaces induced by these norms and a priori estimates of the corresponding regularized estimators.