# Embeddings between Barron spaces with higher order activation functions

arXiv: https://arxiv.org/abs/2305.15839 · HTML: https://mlsanity.com/paper/2305.15839/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 14 of 14)

- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't](https://mlsanity.com/paper/2009.10713/) (2020)
- [Representation Benefits of Deep Feedforward Networks](https://mlsanity.com/paper/1509.08101/) (2015)
- [Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networks](https://mlsanity.com/paper/2101.12365/) (2021)
- [Representation formulas and pointwise properties for Barron functions](https://mlsanity.com/paper/2006.05982/) (2020)
- [Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions](https://mlsanity.com/paper/2006.15459/) (2020)
- [Understanding neural networks with reproducing kernel Banach spaces](https://mlsanity.com/paper/2109.09710/) (2021)
- [Neural network approximation and estimation of classifiers with classification boundary in a Barron class](https://mlsanity.com/paper/2011.09363/) (2020)
- [Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels](https://mlsanity.com/paper/2005.10807/) (2020)
- [Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms](https://mlsanity.com/paper/2009.06132/) (2020)
- [Some observations on high-dimensional partial differential equations with Barron data](https://mlsanity.com/paper/2012.01484/) (2020)
- [Duality for Neural Networks through Reproducing Kernel Banach Spaces](https://mlsanity.com/paper/2211.05020/) (2022)
- [Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks](https://mlsanity.com/paper/2108.12943/) (2021)

## Built on by (most cited first, 4 of 4)

- [A Lipschitz spaces view of infinitely wide shallow neural networks](https://mlsanity.com/paper/2410.14591/) (2024)
- [Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flow](https://mlsanity.com/paper/2312.02671/) (2023)
- [Curse of Dimensionality in Neural Network Optimization](https://mlsanity.com/paper/2502.05360/) (2025)
- [A Dual Certificate Approach to Sparsity in Infinite-Width Shallow Neural Networks](https://mlsanity.com/paper/2603.17785/) (2026)

## Abstract (arXiv)

The approximation properties of infinitely wide shallow neural networks heavily depend on the choice of the activation function. To understand this influence, we study embeddings between Barron spaces with different activation functions. These embeddings are proven by providing push-forward maps on the measures μ used to represent functions f. An activation function of particular interest is the rectified power unit (operatornameRePU) given by operatornameRePUs(x)=max(0,x)s. For many commonly used activation functions, the well-known Taylor remainder theorem can be used to construct a push-forward map, which allows us to prove the embedding of the associated Barron space into a Barron space with a operatornameRePU as activation function. Moreover, the Barron spaces associated with the operatornameRePUs have a hierarchical structure similar to the Sobolev spaces Hm.