# Operator SVD with Neural Networks via Nested Low-Rank Approximation

arXiv: https://arxiv.org/abs/2402.03655 · HTML: https://mlsanity.com/paper/2402.03655/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 98 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 30 of 39)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Ab-Initio Solution of the Many-Electron Schr"odinger Equation with Deep Neural Networks](https://mlsanity.com/paper/1909.02487/) (2019)
- [Deep neural network solution of the electronic Schr"odinger equation](https://mlsanity.com/paper/1909.08423/) (2019)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [A Zero-Shot Framework for Sketch-based Image Retrieval](https://mlsanity.com/paper/1807.11724/) (2018)
- [Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval](https://mlsanity.com/paper/1703.05605/) (2017)
- [Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1903.03372/) (2019)
- [EigenGame: PCA as a Nash Equilibrium](https://mlsanity.com/paper/2010.00554/) (2020)
- [Spectral Inference Networks: Unifying Deep and Spectral Learning](https://mlsanity.com/paper/1806.02215/) (2018)
- [Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approach](https://mlsanity.com/paper/2002.02600/) (2020)
- [Unsupervised Deep Learning Algorithm for PDE-based Forward and Inverse Problems](https://mlsanity.com/paper/1904.05417/) (2019)
- [Ab-initio quantum chemistry with neural-network wavefunctions](https://mlsanity.com/paper/2208.12590/) (2022)
- [NeuralEF: Deconstructing Kernels by Deep Neural Networks](https://mlsanity.com/paper/2205.00165/) (2022)
- [Gold-standard solutions to the Schr"odinger equation using deep learning: How much physics do we need?](https://mlsanity.com/paper/2205.09438/) (2022)
- [Deep K-SVD Denoising](https://mlsanity.com/paper/1909.13164/) (2019)
- [Variational Interaction Information Maximization for Cross-domain Disentanglement](https://mlsanity.com/paper/2012.04251/) (2020)
- [Nonparametric Canonical Correlation Analysis](https://mlsanity.com/paper/1511.04839/) (2015)
- [An Efficient Approach to Informative Feature Extraction from Multimodal Data](https://mlsanity.com/paper/1811.08979/) (2018)
- [Electronic excited states in deep variational Monte Carlo](https://mlsanity.com/paper/2203.09472/) (2022)
- [On Universal Features for High-Dimensional Learning and Inference](https://mlsanity.com/paper/1911.09105/) (2019)
- [An Information Theoretic Interpretation to Deep Neural Networks](https://mlsanity.com/paper/1905.06600/) (2019)
- [Physics-Informed Neural Networks for Quantum Eigenvalue Problems](https://mlsanity.com/paper/2203.00451/) (2022)

## Built on by (most cited first, 13 of 13)

- [Efficient Parametric SVD of Koopman Operator for Stochastic Dynamical Systems](https://mlsanity.com/paper/2507.07222/) (2025)
- [Sequence Modeling with Spectral Mean Flows](https://mlsanity.com/paper/2510.15366/) (2025)
- [STNet: Spectral Transformation Network for Solving Operator Eigenvalue Problem](https://mlsanity.com/paper/2510.23986/) (2025)
- [An effective physics-informed neural operator framework for predicting wavefields](https://mlsanity.com/paper/2507.16431/) (2025)
- [Contrastive Predictive Coding Done Right for Mutual Information Estimation](https://mlsanity.com/paper/2510.25983/) (2025)
- [Accelerating Eigenvalue Dataset Generation via Chebyshev Subspace Filter](https://mlsanity.com/paper/2510.23215/) (2025)
- [Revisiting Orbital Minimization Method for Neural Operator Decomposition](https://mlsanity.com/paper/2510.21952/) (2025)
- [Eigenfunction Extraction for Ordered Representation Learning](https://mlsanity.com/paper/2510.24672/) (2025)
- [Learning-Guided Integration Contours Construction for Fast Large-Scale Generalized Eigensolvers](https://mlsanity.com/paper/2511.01927/) (2025)
- [NeuraLSP: A Neural Spectral Preconditioner for Accelerating PDE Solvers](https://mlsanity.com/paper/2601.20174/) (2026)
- [Minimax Rates and Spectral Distillation for Tree Ensembles](https://mlsanity.com/paper/2605.11841/) (2026)
- [Quasi-SVD: Learning a Lie-constrained matrix factorisation for real-time imaging](https://mlsanity.com/paper/2607.25967/) (2026)
- [Low-Fidelity FDM Spectral Guidance for Neural Eigenvalue Solvers](https://mlsanity.com/paper/2610.04695/) (2026)

## Abstract (arXiv)

Computing eigenvalue decomposition (EVD) of a given linear operator, or finding its leading eigenvalues and eigenfunctions, is a fundamental task in many machine learning and scientific computing problems. For high-dimensional eigenvalue problems, training neural networks to parameterize the eigenfunctions is considered as a promising alternative to the classical numerical linear algebra techniques. This paper proposes a new optimization framework based on the low-rank approximation characterization of a truncated singular value decomposition, accompanied by new techniques called nesting for learning the top-L singular values and singular functions in the correct order. The proposed method promotes the desired orthogonality in the learned functions implicitly and efficiently via an unconstrained optimization formulation, which is easy to solve with off-the-shelf gradient-based optimization algorithms. We demonstrate the effectiveness of the proposed optimization framework for use cases in computational physics and machine learning.