# What if Neural Networks had SVDs?

arXiv: https://arxiv.org/abs/2009.13977 · HTML: https://mlsanity.com/paper/2009.13977/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2020.

## Built on by (most cited first, 8 of 8)

- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [PaTH Attention: Position Encoding via Accumulating Householder Transformations](https://mlsanity.com/paper/2505.16381/) (2025)
- [Principled Multimodal Representation Learning](https://mlsanity.com/paper/2507.17343/) (2025)
- [Neural Gaussian Scale-Space Fields](https://mlsanity.com/paper/2405.20980/) (2024)
- [Householder Projector for Unsupervised Latent Semantics Discovery](https://mlsanity.com/paper/2307.08012/) (2023)
- [OrthCaps: An Orthogonal CapsNet with Sparse Attention Routing and Pruning](https://mlsanity.com/paper/2403.13351/) (2024)
- [Deep Hashing via Householder Quantization](https://mlsanity.com/paper/2311.04207/) (2023)
- [A Robust Prototype-Based Network with Interpretable RBF Classifier Foundations](https://mlsanity.com/paper/2412.15499/) (2024)

## Abstract (arXiv)

Various Neural Networks employ time-consuming matrix operations like matrix inversion. Many such matrix operations are faster to compute given the Singular Value Decomposition (SVD). Previous work allows using the SVD in Neural Networks without computing it. In theory, the techniques can speed up matrix operations, however, in practice, they are not fast enough. We present an algorithm that is fast enough to speed up several matrix operations. The algorithm increases the degree of parallelism of an underlying matrix multiplication H· X where H is an orthogonal matrix represented by a product of Householder matrices. Code is available at www.github.com/AlexanderMath/fasth .