# Singular Value Representation: A New Graph Perspective On Neural Networks

arXiv: https://arxiv.org/abs/2302.08183 · HTML: https://mlsanity.com/paper/2302.08183/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models](https://mlsanity.com/paper/1704.01942/) (2017)
- [Introduction to Tensor Decompositions and their Applications in Machine Learning](https://mlsanity.com/paper/1711.10781/) (2017)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Coordinating Filters for Faster Deep Neural Networks](https://mlsanity.com/paper/1703.09746/) (2017)
- [Neural Teleportation](https://mlsanity.com/paper/2012.01118/) (2020)
- [On Implicit Filter Level Sparsity in Convolutional Neural Networks](https://mlsanity.com/paper/1811.12495/) (2018)
- [Projected distances for multi-parameter persistence modules](https://mlsanity.com/paper/2206.08818/) (2022)

## Built on by (most cited first, 3 of 3)

- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Hyper-Representations: Learning from Populations of Neural Networks](https://mlsanity.com/paper/2410.05107/) (2024)
- [Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space](https://mlsanity.com/paper/2609.37731/) (2026)

## Abstract (arXiv)

We introduce the Singular Value Representation (SVR), a new method to represent the internal state of neural networks using SVD factorization of the weights. This construction yields a new weighted graph connecting what we call spectral neurons, that correspond to specific activation patterns of classical neurons. We derive a precise statistical framework to discriminate meaningful connections between spectral neurons for fully connected and convolutional layers. To demonstrate the usefulness of our approach for machine learning research, we highlight two discoveries we made using the SVR. First, we highlight the emergence of a dominant connection in VGG networks that spans multiple deep layers. Second, we witness, without relying on any input data, that batch normalization can induce significant connections between near-kernels of deep layers, leading to a remarkable spontaneous sparsification phenomenon.