mlsanityData to 7 Oct 2026

ResNet with one-neuron hidden layers is a Universal Approximator

arXiv 1806.10909 ↗First posted 28 Jun 2018cs.LG stat.MLcited by 78

Not on the map. Why?

Abstract (arXiv)

We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in d dimensions, i.e. ℓ1(ℝd). Because of the identity mapping inherent to ResNets, our network has alternating layers of dimension one and d. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension d [Lu et al, 2017; Hanin and Sellke, 2017]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 17; 2023: 10; 2024: 8; 2025: 11; 2026: 14.

78 papers.

Sort:

By area:

  1. 2021Choose a Transformer: Fourier or Galerkincited by 333
  2. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  3. 2021Neural Flows: Efficient Alternative to Neural ODEscited by 99
  4. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  5. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  6. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 46
  7. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  8. 2022Residual-based error correction for neural operator accelerated infinite-dimensional Bayesian inverse problemscited by 21
  9. 2021Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Seriescited by 16
  10. 2021Learning High-Dimensional Parametric Maps via Reduced Basis Adaptive Residual Networkscited by 16
  11. 2022Approximation of Lipschitz Functions using Deep Spline Neural Networkscited by 15
  12. 2024A Survey on Universal Approximation Theoremscited by 15
38 more
  1. 2021Understanding the computational demands underlying visual reasoningcited by 13
  2. 2022Robust Classification using Contractive Hamiltonian Neural ODEscited by 13
  3. 2022Minimal Width for Universal Property of Deep RNNcited by 12
  4. 2024Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networkscited by 11
  5. 2024PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videoscited by 10
  6. 2022On Measuring Excess Capacity in Neural Networkscited by 9
  7. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  8. 2024Interplay between depth and width for interpolation in neural ODEscited by 8
  9. 2021Universal Approximation of Residual Flows in Maximum Mean Discrepancycited by 7
  10. 2021Meta Internal Learningcited by 7
  11. 2022Piecewise Linear Neural Networks and Deep Learningcited by 7
  12. 2022The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Networkcited by 7
  13. 2021An Interpretive Constrained Linear Model for ResNet and MgNetcited by 6
  14. 2023Tailor: Altering Skip Connections for Resource-Efficient Inferencecited by 6
  15. 2022Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systemscited by 4
  16. 2023Cluster-based classification with neural ODEs via controlcited by 4
  17. 2021K"ahler Geometry of Quiver Varieties and Machine Learningcited by 3
  18. 2025A Dynamical Systems Perspective on the Analysis of Neural Networkscited by 3
  19. 2025Some Super-approximation Rates of ReLU Neural Networks for Korobov Functionscited by 3
  20. 2021Reconstructing a dynamical system and forecasting time series by self-consistent deep learningcited by 2
  21. 2021Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networkscited by 2
  22. 2022Modelling of physical systems with a Hopf bifurcation using mechanistic models and machine learningcited by 2
  23. 2023Universal Approximation Property of Hamiltonian Deep Neural Networkscited by 2
  24. 2023Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networkscited by 2
  25. 2023A Minimal Control Family of Dynamical Systems for Universal Approximationcited by 2
  26. 2024Disparate Impact on Group Accuracy of Linearization for Private Inferencecited by 2
  27. 2024Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machinescited by 2
  28. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 2
  29. 2025Hybrid Least Squares/Gradient Descent Methods for DeepONetscited by 2
  30. 2025Efficient Manifold-Constrained Neural ODE for High-Dimensional Datasetscited by 2
  31. 2025FlowPath: Learning Data-Driven Manifolds with Invertible Flows for Robust Irregularly-sampled Time Series Classificationcited by 2
  32. 2025Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encodingcited by 2
  33. 2021Subspace Clustering Based Analysis of Neural Networkscited by 1
  34. 2022Convergence Analysis of Deep Residual Networkscited by 1
  35. 2022Neural interval-censored survival regression with feature selectioncited by 1
  36. 2022Neural Network Approximation of Continuous Functions in High Dimensions with Applications to Inverse Problemscited by 1
  37. 2022LU decomposition and Toeplitz decomposition of a neural networkcited by 1
  38. 2024Demystifying the Token Dynamics of Deep Selective State Space Modelscited by 1

50 most cited of 78 All 78 (JSON)