# Elementary superexpressive activations

arXiv: https://arxiv.org/abs/2102.10911 · HTML: https://mlsanity.com/paper/2102.10911/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2021.

## Built on by (most cited first, 30 of 36)

- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Operator Learning: Algorithms and Analysis](https://mlsanity.com/paper/2402.15715/) (2024)
- [Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces](https://mlsanity.com/paper/2201.00217/) (2022)
- [Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons](https://mlsanity.com/paper/2107.02397/) (2021)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [Deep Operator Network Approximation Rates for Lipschitz Operators](https://mlsanity.com/paper/2307.09835/) (2023)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Finite Expression Method for Solving High-Dimensional Partial Differential Equations](https://mlsanity.com/paper/2206.10121/) (2022)
- [Deep Operator Learning Lessens the Curse of Dimensionality for PDEs](https://mlsanity.com/paper/2301.12227/) (2023)
- [Mixtures of Neural Operators Reduce Active Complexity in Operator Learning](https://mlsanity.com/paper/2404.09101/) (2024)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [Deep Network Approximation in Terms of Intrinsic Parameters](https://mlsanity.com/paper/2111.07964/) (2021)
- [Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions](https://mlsanity.com/paper/2312.14276/) (2023)
- [Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Class](https://mlsanity.com/paper/2103.00542/) (2021)
- [Noncompact uniform universal approximation](https://mlsanity.com/paper/2308.03812/) (2023)
- [Don't Fear Peculiar Activation Functions: EUAF and Beyond](https://mlsanity.com/paper/2407.09580/) (2024)
- [Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?](https://mlsanity.com/paper/2204.11231/) (2022)
- [Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective](https://mlsanity.com/paper/2504.13558/) (2025)
- [Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential](https://mlsanity.com/paper/2502.18959/) (2025)
- [Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections](https://mlsanity.com/paper/2504.15110/) (2025)
- [Neural networks with superexpressive activations and integer weights](https://mlsanity.com/paper/2105.09917/) (2021)
- [Deep ReLU networks and high-order finite element methods II: Chebyshev emulation](https://mlsanity.com/paper/2310.07261/) (2023)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Optimal Neural Network Approximation for High-Dimensional Continuous Functions](https://mlsanity.com/paper/2409.02363/) (2024)
- [Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data](https://mlsanity.com/paper/2509.00924/) (2025)
- [Layer-wise Geometric Approximation Rates for Deep Networks](https://mlsanity.com/paper/2604.20219/) (2026)
- [Structure of universal formulas](https://mlsanity.com/paper/2311.03910/) (2023)
- [Neural Network Approximation for Pessimistic Offline Reinforcement Learning](https://mlsanity.com/paper/2312.11863/) (2023)
- [Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method](https://mlsanity.com/paper/2405.11451/) (2024)
- [Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations](https://mlsanity.com/paper/2503.21166/) (2025)

## Abstract (arXiv)

We call a finite family of activation functions superexpressive if any multivariate continuous function can be approximated by a neural network that uses these activations and has a fixed architecture only depending on the number of input variables (i.e., to achieve any accuracy we only need to adjust the weights, without increasing the number of neurons). Previously, it was known that superexpressive activations exist, but their form was quite complex. We give examples of very simple superexpressive families: for example, we prove that the family {sin, arcsin} is superexpressive. We also show that most practical activations (not involving periodic functions) are not superexpressive.