mlsanityData to 14 Oct 2026

Elementary superexpressive activations

arXiv 2102.10911 ↗First posted 22 Feb 2021cs.NEcited by 36

Not on the map. Why?

Abstract (arXiv)

We call a finite family of activation functions superexpressive if any multivariate continuous function can be approximated by a neural network that uses these activations and has a fixed architecture only depending on the number of input variables (i.e., to achieve any accuracy we only need to adjust the weights, without increasing the number of neurons). Previously, it was known that superexpressive activations exist, but their form was quite complex. We give examples of very simple superexpressive families: for example, we prove that the family {sin, arcsin} is superexpressive. We also show that most practical activations (not involving periodic functions) are not superexpressive.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Feb 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 4; 2023: 8; 2024: 7; 2025: 7; 2026: 4.

36 papers.

Sort:

By area:

  1. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  2. 2024Operator Learning: Algorithms and Analysiscited by 85
  3. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  4. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  5. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  6. 2023Deep Operator Network Approximation Rates for Lipschitz Operatorscited by 24
  7. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  8. 2022Finite Expression Method for Solving High-Dimensional Partial Differential Equationscited by 21
  9. 2023Deep Operator Learning Lessens the Curse of Dimensionality for PDEscited by 18
  10. 2024Mixtures of Neural Operators Reduce Active Complexity in Operator Learningcited by 16
  11. 2023On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theoremcited by 13
  12. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
24 more
  1. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8
  2. 2021Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Classcited by 7
  3. 2023Noncompact uniform universal approximationcited by 7
  4. 2024Don't Fear Peculiar Activation Functions: EUAF and Beyondcited by 7
  5. 2022Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?cited by 6
  6. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  7. 2025Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potentialcited by 4
  8. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 4
  9. 2021Neural networks with superexpressive activations and integer weightscited by 3
  10. 2023Deep ReLU networks and high-order finite element methods II: Chebyshev emulationcited by 3
  11. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Expertscited by 2
  12. 2024Optimal Neural Network Approximation for High-Dimensional Continuous Functionscited by 2
  13. 2025Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Datacited by 2
  14. 2026Layer-wise Geometric Approximation Rates for Deep Networkscited by 2
  15. 2023Structure of universal formulascited by 1
  16. 2023Neural Network Approximation for Pessimistic Offline Reinforcement Learningcited by 1
  17. 2024Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Methodcited by 1
  18. 2025Unveiling the Potential of Superexpressive Networks in Implicit Neural Representationscited by 1
  19. 2025Upper Approximation Bounds for Neural Oscillatorscited by 1
  20. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1
  21. 2024Learnability of high-dimensional targets by two-parameter models and gradient flowcited by 0
  22. 2025NeuronSeek: On Stability and Expressivity of Task-driven Neuronscited by 0
  23. 2026Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexitycited by 0
  24. 2026Description Complexity Bounds for Analytic Approximations of Lipschitz Functionscited by 0