mlsanityData to 14 Oct 2026

A Nonlinear Singular Value Theory for Neural Networks

arXiv 2605.06938 ↗First posted 7 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recently Brown et al. [2025] established a singular value decomposition (SVD) for maps (especially nonlinear) satisfying certain norm conditions. We prove that most modern neural architectures admit this nonlinear SVD (NLSVD) representation---with no change in input--output behavior---and enumerate the classes covered. In this factorization the network is a left-invertible nonlinear map followed by a final linear layer. Moreover, the left-invertible factor is norm-preserving, so distances in the embedding (activations before the final linear layer) calibrate directly to distances in input space. We introduce a flexible architecture that yields an explicit decomposition at training time, a data-driven algorithm for estimating the representation from trained models, and the mathematical foundations for nonlinear analogues of row and null spaces in neural networks. Empirical case studies illustrate uses of the theory for latent-space pullback (visualization and data generation), bias detection, and membership-inference robustness under training. Altogether, these foundations support new approaches to core problems in neural-network analysis.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

18 of 29 references.

References · 18

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  7. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  8. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  9. 2023Steering Language Models With Activation Engineeringcited by 922
  10. 2021Membership Inference Attacks From First Principlescited by 878
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
6 more
  1. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  2. 2018Invertible Residual Networkscited by 365
  3. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  4. 2020The Lipschitz Constant of Self-Attentioncited by 174
  5. 2018The Singular Values of Convolutional Layerscited by 95
  6. 2023Manifold learning: what, how, and whycited by 68

Built on by

Not cited yet.