Structured and Balanced Multi-Component and Multi-Layer Neural Networks
Not on the map. Why?
Abstract (arXiv)
In this work, we propose a balanced multi-component and multi-layer neural network (MMNN) structure to accurately and efficiently approximate functions with complex features, in terms of both degrees of freedom and computational cost. The main idea is inspired by a multi-component approach, in which each component can be effectively approximated by a single-layer network, combined with a multi-layer decomposition strategy to capture the complexity of the target function. Although MMNNs can be viewed as a simple modification of fully connected neural networks (FCNNs) or multi-layer perceptrons (MLPs) by introducing balanced multi-component structures, they achieve a significant reduction in training parameters, a much more efficient training process, and improved accuracy compared to FCNNs or MLPs. Extensive numerical experiments demonstrate the effectiveness of MMNNs in approximating highly oscillatory functions and their ability to automatically adapt to localized features.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 30 Jun 2024.
Builds on
31 of 48 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2024KAN: Kolmogorov-Arnold Networkscited by 634
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2021Random Feature Attentioncited by 350
- 2015Tensorizing Neural Networkscited by 246
- 2018Universality of Deep Convolutional Neural Networkscited by 176
- 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
- 2016Generalization Properties of Learning with Random Featurescited by 163
- 2020Deep Network Approximation for Smooth Functionscited by 163
19 more
- 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
- 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 107
- 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
- 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
- 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
- 2019On the Existence of Simpler Machine Learning Modelscited by 87
- 2019Approximation spaces of deep neural networkscited by 51
- 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
- 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 45
- 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
- 2019Nonlinear Approximation via Compositionscited by 37
- 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
- 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
- 2023On the Kolmogorov neural networkscited by 19
- 2022Neural Network Architecture Beyond Width and Depthcited by 17
- 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
- 2023Why Shallow Networks Struggle to Approximate and Learn High Frequenciescited by 10
- 2018Construction of neural networks for realization of localized deep learningcited by 7
- 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.