mlsanityData to 14 Oct 2026

Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?

arXiv 2204.11231 ↗First posted 24 Apr 2022cs.LG cs.AI cs.NAcited by 6

Not on the map. Why?

Abstract (arXiv)

We study the problem of approximating compactly-supported integrable functions while implementing their support set using feedforward neural networks. Our first main result transcribes this "structured" approximation problem into a universality problem. We do this by constructing a refinement of the usual topology on the space L1_operatornameloc(ℝd,ℝD) of locally-integrable functions in which compactly-supported functions can only be approximated in L1-norm by functions with matching discretized support. We establish the universality of ReLU feedforward networks with bilinear pooling layers in this refined topology. Consequentially, we find that ReLU feedforward networks with bilinear pooling can approximate compactly supported functions while implementing their discretized support. We derive a quantitative uniform version of our universal approximation theorem on the dense subclass of compactly-supported Lipschitz functions. This quantitative result expresses the depth, width, and the number of bilinear pooling layers required to construct this ReLU network via the target function's regularity, the metric capacity and diameter of its essential support, and the dimensions of the inputs and output spaces. Conversely, we show that polynomial regressors and analytic feedforward networks are not universal in this space.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 34 of its 79 references that are in the map. First posted on arXiv 24 Apr 2022.

Builds on

34 of 79 references.

References · 34

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  5. 2016Error bounds for approximations with deep ReLU networkscited by 574
  6. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  7. 2015Bilinear CNNs for Fine-grained Visual Recognitioncited by 228
  8. 2019Universal Approximation with Deep Narrow Networkscited by 200
  9. 2018Universality of Deep Convolutional Neural Networkscited by 176
  10. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 166
  11. 2020Deep Network Approximation for Smooth Functionscited by 164
  12. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
22 more
  1. 2018Universal approximations of invariant maps by neural networkscited by 127
  2. 2018Deep Hedgingcited by 108
  3. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  4. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  5. 2020Minimum Width for Universal Approximationcited by 94
  6. 2019The phase diagram of approximation rates for deep neural networkscited by 88
  7. 2021On the approximation of functions by tanh neural networkscited by 81
  8. 2021The Old and the New: Can Physics-Informed Deep-Learning Replace Traditional Linear Solvers?cited by 61
  9. 2019Deep splitting method for parabolic PDEscited by 53
  10. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
  11. 2019Approximation spaces of deep neural networkscited by 51
  12. 2018Solving the Kolmogorov PDE by means of deep learningcited by 41
  13. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  14. 2021Elementary superexpressive activationscited by 36
  15. 2020Physics Informed Neural Networks for Simulating Radiative Transfercited by 35
  16. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  17. 2020Non-Euclidean Universal Approximationcited by 29
  18. 2019Differentiable reservoir computingcited by 20
  19. 2021Universal Joint Approximation of Manifolds and Densities by Simple Injective Flowscited by 13
  20. 2018NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representationcited by 6
  21. 2020Learning Sub-Patterns in Piecewise Continuous Functionscited by 4
  22. 2021Neural networks with superexpressive activations and integer weightscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 2; 2026: 1.

6 papers.

  1. 2022Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysiscited by 22
  2. 2025One model to solve them all: 2BSDE families via neural operatorscited by 4
  3. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Expertscited by 2
  4. 2023Data Topology-Dependent Upper Bounds of Neural Network Widthscited by 1
  5. 2025Autonomous-Flow-Based Generationcited by 1
  6. 2026Structure-Preserving Reconstruction of Convex Lipschitz Functionals on Hilbert Spaces from Finite Samplescited by 0