mlsanityData to 7 Oct 2026

Spectral Neural Networks: Approximation Theory and Optimization Landscape

arXiv 2310.00729 ↗First posted 1 Oct 2023cs.LG cs.NA math.APcited by 2

Not on the map. Why?

Abstract (arXiv)

There is a large variety of machine learning methodologies that are based on the extraction of spectral geometric information from data. However, the implementations of many of these methods often depend on traditional eigensolvers, which present limitations when applied in practical online big data scenarios. To address some of these challenges, researchers have proposed different strategies for training neural networks as alternatives to traditional eigensolvers, with one such approach known as Spectral Neural Network (SNN). In this paper, we investigate key theoretical aspects of SNN. First, we present quantitative insights into the tradeoff between the number of neurons and the amount of spectral geometric information a neural network learns. Second, we initiate a theoretical exploration of the optimization landscape of SNN's objective to shed light on the training dynamics of SNN. Unlike typical studies of convergence to global solutions of NN training dynamics, SNN presents an additional complexity due to its non-convex ambient loss function.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 69 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

38 of 69 references.

References · 38

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  7. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  8. 2019Benign Overfitting in Linear Regressioncited by 544
  9. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  10. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  12. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
26 more
  1. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  2. 2018Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantificationcited by 199
  3. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  4. 2020Deep Network Approximation for Smooth Functionscited by 163
  5. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  6. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  7. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  8. 2017Solving ill-posed inverse problems using iterative deep neural networkscited by 145
  9. 2019Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networkscited by 144
  10. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  11. 2018SpectralNet: Spectral Clustering using Deep Neural Networkscited by 94
  12. 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 76
  13. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  14. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  15. 2018Error estimates for spectral convergence of the graph Laplacian on random geometric graphs towards the Laplace--Beltrami operatorcited by 70
  16. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  17. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  18. 2019Nonlinear Approximation via Compositionscited by 37
  19. 2015A variational approach to the consistency of spectral clusteringcited by 28
  20. 2020Lipschitz regularity of graph Laplacians on random data cloudscited by 26
  21. 2020Spectral convergence of diffusion maps: improved error bounds and an alternative normalisationcited by 25
  22. 2023Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoiserscited by 13
  23. 2023Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Losscited by 7
  24. 2016The Non-convex Geometry of Low-rank Matrix Optimizationcited by 6
  25. 2021On Geometric Connections of Embedded and Quotient Geometries in Riemannian Fixed-rank Matrix Optimizationcited by 3
  26. 2022Nonconvex Matrix Factorization is Geodesically Convex: Global Landscape Analysis for Fixed-rank Matrix Optimization From a Riemannian Perspectivecited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Quantifying Structure in CLIP Embeddings: A Statistical Framework for Concept Interpretationcited by 6
  2. 2025S3F-Net: A Multi-Modal Approach to Medical Image Classification via Spatial-Spectral Summarizer Fusion Networkcited by 0