# Low-Rank Matrix Approximation for Neural Network Compression

arXiv: https://arxiv.org/abs/2504.20078 · HTML: https://mlsanity.com/paper/2504.20078/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 19 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Robust Non-Linear Matrix Factorization for Dictionary Learning, Denoising, and Clustering](https://mlsanity.com/paper/2005.01317/) (2020)
- [A Computational-Graph Partitioning Method for Training Memory-Constrained DNNs](https://mlsanity.com/paper/2008.08636/) (2020)

## Built on by (most cited first, 3 of 3)

- [Rank Is Not Capacity: Spectral Occupancy for Latent Graph Models](https://mlsanity.com/paper/2605.11142/) (2026)
- [Integrating Pruning with Quantization for Efficient Deep Neural Networks Compression](https://mlsanity.com/paper/2509.04244/) (2025)
- [Noise-robust Contrastive Learning for Critical Transition Detection in Dynamical Systems](https://mlsanity.com/paper/2512.12523/) (2025)

## Abstract (arXiv)

Deep Neural Networks (DNNs) have encountered an emerging deployment challenge due to large and expensive memory and computation requirements. In this paper, we present a new Adaptive-Rank Singular Value Decomposition (ARSVD) method that approximates the optimal rank for compressing weight matrices in neural networks using spectral entropy. Unlike conventional SVD-based methods that apply a fixed-rank truncation across all layers, ARSVD uses an adaptive selection of the rank per layer through the entropy distribution of its singular values. This approach ensures that each layer will retain a certain amount of its informational content, thereby reducing redundancy. Our method enables efficient, layer-wise compression, yielding improved performance with reduced space and time complexity compared to static-rank reduction techniques.