# Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations

arXiv: https://arxiv.org/abs/2512.12132 · HTML: https://mlsanity.com/paper/2512.12132/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 13 Dec 2025.

## Builds on (most cited first, 9 of 9)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Neural Networks Learning and Memorization with (almost) no Over-Parameterization](https://mlsanity.com/paper/1911.09873/) (2019)
- [GELU Activation Function in Deep Learning: A Comprehensive Mathematical Analysis and Performance](https://mlsanity.com/paper/2305.12073/) (2023)

## Abstract (arXiv)

We present SiLU network constructions whose approximation efficiency depends critically on proper hyperparameter tuning. For the square function x2, with optimally chosen shift a and scale β, we achieve approximation error varepsilon using a two-layer network of constant width, where weights scale as β± k with k = O(ln(1/varepsilon)). We then extend this approach through functional composition to Sobolev spaces, we obtain networks with depth O(1) and O(varepsilon-d/n) parameters under optimal hyperparameters settings. Our work highlights the trade-off between architectural depth and activation parameter optimization in neural network approximation theory.