mlsanityData to 7 Oct 2026

Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions

arXiv 2301.04605 ↗First posted 11 Jan 2023cs.LG cs.NE math.FAcited by 4

Not on the map. Why?

Abstract (arXiv)

Multiplication layers are a key component in various influential neural network modules, including self-attention and hypernetwork layers. In this paper, we investigate the approximation capabilities of deep neural networks with intermediate neurons connected by simple multiplication operations. We consider two classes of target functions: generalized bandlimited functions, which are frequently used to model real-world signals with finite bandwidth, and Sobolev-Type balls, which are embedded in the Sobolev Space Wr,2. Our results demonstrate that multiplicative neural networks can approximate these functions with significantly fewer layers and neurons compared to standard ReLU neural networks, with respect to both input dimension and approximation error. These findings suggest that multiplicative gates can outperform standard feed-forward layers and have potential for improving neural network design.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 88 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

33 of 88 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2016HyperNetworkscited by 1,003
  9. 2021Scaling Vision Transformerscited by 993
  10. 2016Error bounds for approximations with deep ReLU networkscited by 567
  11. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  12. 2017Stable Architectures for Deep Neural Networkscited by 340
21 more
  1. 2019Continual learning with hypernetworkscited by 267
  2. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  3. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  4. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  5. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  6. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  7. 2021PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equationscited by 106
  8. 2016Why Deep Neural Networks for Function Approximation?cited by 82
  9. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  10. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  11. 2018Neural Arithmetic Logic Unitscited by 69
  12. 2017Bayesian Hypernetworkscited by 64
  13. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  14. 2019Deep Meta Functionals for Shape Representationcited by 53
  15. 2019AlphaStar: An Evolutionary Computation Perspectivecited by 50
  16. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  17. 2020The Depth-to-Width Interplay in Self-Attentioncited by 39
  18. 2016Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networkscited by 26
  19. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  20. 2021Meta Internal Learningcited by 7
  21. 2015A Neural Transfer Function for a Smooth and Differentiable Transition Between Additive and Multiplicative Interactionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 1.

4 papers.

  1. 2023Feature Transportation Improves Graph Neural Networkscited by 15
  2. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  3. 2025A New Perspective To Understanding Multi-resolution Hash Encoding For Neural Fieldscited by 2
  4. 2026(MPO)2: Multivariate Polynomial Optimization based on Matrix Product Operatorscited by 0