mlsanityData to 7 Oct 2026

Integral Signatures of Activation Functions: A 9-Dimensional Taxonomy and Stability Theory for Deep Learning

arXiv 2510.08456 ↗First posted 9 Oct 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Activation functions govern the expressivity and stability of neural networks, yet existing comparisons remain largely heuristic. We propose a rigorous framework for their classification via a nine-dimensional integral signature S_sigma(phi), combining Gaussian propagation statistics (m1, g1, g2, m2, eta), asymptotic slopes (alpha_plus, alpha_minus), and regularity measures (TV(phi'), C(phi)). This taxonomy establishes well-posedness, affine reparameterization laws with bias, and closure under bounded slope variation. Dynamical analysis yields Lyapunov theorems with explicit descent constants and identifies variance stability regions through (m2', g2). From a kernel perspective, we derive dimension-free Hessian bounds and connect smoothness to bounded variation of phi'. Applying the framework, we classify eight standard activations (ReLU, leaky-ReLU, tanh, sigmoid, Swish, GELU, Mish, TeLU), proving sharp distinctions between saturating, linear-growth, and smooth families. Numerical Gauss-Hermite and Monte Carlo validation confirms theoretical predictions. Our framework provides principled design guidance, moving activation choice from trial-and-error to provable stability and kernel conditioning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 35 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

19 of 35 references.

References · 19

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  8. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  9. 2016Deep Information Propagationcited by 194
  10. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  11. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  12. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
7 more
  1. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  2. 2020A survey on modern trainable activation functionscited by 89
  3. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  4. 2020Rational neural networkscited by 75
  5. 2019Smooth function approximation by deep neural networks with general activation functionscited by 29
  6. 2020Approximating smooth functions by deep neural networks with sigmoid activation functioncited by 18
  7. 2024TeLU Activation Function for Fast and Stable Deep Learningcited by 4

Built on by

Not cited yet.