mlsanityData to 14 Oct 2026

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

arXiv 2502.18959 ↗First posted 26 Feb 2025cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

The architecture of a neural network and the choice of its activation function are both fundamental to its performance. Equally important is ensuring that these two elements are well matched, as their alignment is key to effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a model that combines sine-type activations with the multi-component and multi-layer structure of MMNNs. In an FMMNN, each component is represented as a trainable linear combination of fixed random sine-type basis functions, while multi-layer composition generates more complex and adaptive high-frequency features. We establish that FMMNNs retain exponential expressive power for function approximation even under a low-rank architectural structure. We also analyze the optimization landscape of FMMNNs and find it to be substantially more favorable than that of standard fully connected neural networks, especially for high-frequency targets. In addition, we propose a scaled random initialization method for the first-layer weights in FMMNNs, which accelerates training and improves final performance when sufficient samples are available. Extensive numerical experiments support our theoretical insights, showing that FMMNNs achieve strong accuracy and favorable convergence behavior on oscillatory function-approximation benchmarks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

36 of 51 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  7. 2018On the Spectral Bias of Neural Networkscited by 1,341
  8. 2016Error bounds for approximations with deep ReLU networkscited by 574
  9. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 368
  10. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 284
  11. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  12. 2018Universality of Deep Convolutional Neural Networkscited by 176
24 more
  1. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 166
  2. 2020Deep Network Approximation for Smooth Functionscited by 164
  3. 2019Deep Network Approximation Characterized by Number of Neuronscited by 122
  4. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
  5. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  6. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 91
  7. 2019The phase diagram of approximation rates for deep neural networkscited by 88
  8. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 79
  9. 2023FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functionscited by 78
  10. 2019Approximation spaces of deep neural networkscited by 51
  11. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  12. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  13. 2019Nonlinear Approximation via Compositionscited by 38
  14. 2021Elementary superexpressive activationscited by 36
  15. 2019A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problemscited by 35
  16. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  17. 2019Multi-scale Deep Neural Networks for Solving High Dimensional PDEscited by 25
  18. 2024Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learningcited by 14
  19. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  20. 2023Why Shallow Networks Struggle to Approximate and Learn High Frequenciescited by 11
  21. 2024FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representationcited by 11
  22. 2018Construction of neural networks for realization of localized deep learningcited by 7
  23. 2021Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Classcited by 7
  24. 2024Structured and Balanced Multi-Component and Multi-Layer Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Multigrade Neural Network Approximationcited by 2
  2. 2026From Sparse Sensors to Continuous Fields: STRIDE for Spatiotemporal Reconstructioncited by 2
  3. 2026Layer-wise Geometric Approximation Rates for Deep Networkscited by 2
  4. 2026Solving and learning advective multiscale Darcian dynamics with the Neural Basis Methodcited by 0