mlsanityData to 7 Oct 2026

Don't Fear Peculiar Activation Functions: EUAF and Beyond

arXiv 2407.09580 ↗First posted 12 Jul 2024cs.CV cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a new super-expressive activation function called the Parametric Elementary Universal Activation Function (PEUAF). We demonstrate the effectiveness of PEUAF through systematic and comprehensive experiments on various industrial and image datasets, including CIFAR10, Tiny-ImageNet, and ImageNet. Moreover, we significantly generalize the family of super-expressive activation functions, whose existence has been demonstrated in several recent works by showing that any continuous function can be approximated to any desired accuracy by a fixed-size network with a specific super-expressive activation function. Specifically, our work addresses two major bottlenecks in impeding the development of super-expressive activation functions: the limited identification of super-expressive functions, which raises doubts about their broad applicability, and their often peculiar forms, which lead to skepticism regarding their scalability and practicality in real-world applications.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 26 of its 52 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

26 of 52 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  11. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  12. 2014Striving for Simplicity: The All Convolutional Netcited by 983
14 more
  1. 2017Self-Normalizing Neural Networkscited by 591
  2. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  3. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  4. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  5. 2020A survey on modern trainable activation functionscited by 89
  6. 2021AutoMix: Unveiling the Power of Mixup for Stronger Classifierscited by 69
  7. 2020Neural Network Approximation: Three Hidden Layers Are Enoughcited by 63
  8. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  9. 2021Elementary superexpressive activationscited by 35
  10. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  11. 2020Discovering Parametric Activation Functionscited by 21
  12. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  13. 2023Data-aware customization of activation functions reduces neural network errorcited by 1
  14. 2024Pivotal Auto-Encoder via Self-Normalizing ReLUcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  2. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 2
  3. 2024Optimal Neural Network Approximation for High-Dimensional Continuous Functionscited by 1
  4. 2026Multigrade Neural Network Approximationcited by 1
  5. 2026Layer-wise Geometric Approximation Rates for Deep Networkscited by 1
  6. 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Conditioncited by 0
  7. 2026On Explicit Super-Expressive Approximation for Neural Networkscited by 0