# Exponential Approximation Rates and Parameter Efficiency of Learnable Bernstein Activations

arXiv: https://arxiv.org/abs/2602.04264 · HTML: https://mlsanity.com/paper/2602.04264/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 6 of 6)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [On the Expressive Power of Deep Polynomial Neural Networks](https://mlsanity.com/paper/1905.12207/) (2019)
- [Neural Network Approximation: Three Hidden Layers Are Enough](https://mlsanity.com/paper/2010.14075/) (2020)
- [SELU: Self-Learning Embodied MLLMs in Unknown Environments](https://mlsanity.com/paper/2410.03303/) (2024)
- [DeepBern-Nets: Taming the Complexity of Certifying Neural Networks using Bernstein Polynomial Activations and Precise Bound Propagation](https://mlsanity.com/paper/2305.13508/) (2023)

## Abstract (arXiv)

The choice of activation function fundamentally shapes the representational capacity and parameter efficiency of deep neural networks, yet most widely used activations lack rigorous theoretical guarantees on these properties. We provide a theoretical analysis of DeepBern-Nets (DBNs) -- networks employing learnable Bernstein polynomial activations -- showing that their approximation error decays with the network depth L and the polynomial order n with a rate of O(n-L), exponentially faster than the polynomial rate of ReLU architectures while remaining fully differentiable. We validate these predictions through 1,344 experiments on large scientific datasets (HIGGS and SUSY), comparing DBNs against ReLU, Leaky ReLU, SELU, and GeLU. DBNs achieve over 70% parameter reduction across the majority of architectures -- reaching 99.9% at scale -- converge to ReLU's final loss in as few as 26% of the training epochs, and attain up to 45% lower final loss. These advantages hold over all tested activations, confirming that DBN's gains stem from the learnable polynomial structure rather than mere smoothness.