mlsanityData to 14 Oct 2026

Minimum width for universal approximation using squashable activation functions

arXiv 2504.07371 ↗First posted 10 Apr 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The exact minimum width that allows for universal approximation of unbounded-depth networks is known only for ReLU and its variants. In this work, we study the minimum width of networks using general activation functions. Specifically, we focus on squashable functions that can approximate the identity function and binary step function by alternatively composing with affine transformations. We show that for networks using a squashable activation function to universally approximate Lp functions from [0,1]dx to mathbb Rdy, the minimum width is maxdx,dy,2 unless dx=dy=1; the same bound holds for dx=dy=1 if the activation function is monotone. We then provide sufficient conditions for squashability and show that all non-affine analytic functions and a class of piecewise functions are squashable, i.e., our minimum width result holds for those general classes of activation functions.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 12 of its 21 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

12 of 21 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  4. 2019Universal Approximation with Deep Narrow Networkscited by 200
  5. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 166
  6. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 122
  7. 2020Minimum Width for Universal Approximationcited by 94
  8. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  9. 2018Deep, Skinny Neural Networks are not Universal Approximatorscited by 38
  10. 2020Provable Memorization via Deep Neural Networks using Sub-linear Parameterscited by 34
  11. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 26
  12. 2023Minimum width for universal approximation using ReLU networks on compact domaincited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Parallel Layer Normalization for Universal Approximationcited by 0
  2. 2026Sparse FEONet: A Low-Cost, Memory-Efficient Operator Network via Finite-Element Local Sparsity for Parametric PDEscited by 0
  3. 2026The minimal width of neural networks and immersions of real projective spacecited by 0