# Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation

arXiv: https://arxiv.org/abs/2009.05647 · HTML: https://mlsanity.com/paper/2009.05647/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2020.

## Built on by (most cited first, 8 of 8)

- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [Implicit Regularization Towards Rank Minimization in ReLU Networks](https://mlsanity.com/paper/2201.12760/) (2022)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods](https://mlsanity.com/paper/2402.04051/) (2024)
- [TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNT](https://mlsanity.com/paper/2104.02233/) (2021)
- [Revealing the Utilized Rank of Subspaces of Learning in Neural Networks](https://mlsanity.com/paper/2407.04797/) (2024)
- [DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights](https://mlsanity.com/paper/2501.18596/) (2025)
- [BERMo: What can BERT learn from ELMo?](https://mlsanity.com/paper/2110.15802/) (2021)

## Abstract (arXiv)

A common technique for compressing a neural network is to compute the k-rank ℓ2 approximation Ak,2 of the matrix A∈ℝn× d that corresponds to a fully connected layer (or embedding layer). Here, d is the number of the neurons in the layer, n is the number in the next one, and Ak,2 can be stored in O((n+d)k) memory instead of O(nd). This ℓ2-approximation minimizes the sum over every entry to the power of p=2 in the matrix A - Ak,2, among every matrix Ak,2∈ℝn× d whose rank is k. While it can be computed efficiently via SVD, the ℓ2-approximation is known to be very sensitive to outliers ("far-away" rows). Hence, machine learning uses e.g. Lasso Regression, ℓ1-regularization, and ℓ1-SVM that use the ℓ1-norm. This paper suggests to replace the k-rank ℓ2 approximation by ℓp, for p∈ [1,2]. We then provide practical and provable approximation algorithms to compute it for any p≥1, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage. For example, our approach achieves 28% compression of RoBERTa's embedding layer with only 0.63% additive drop in the accuracy (without fine-tuning) in average over all tasks in GLUE, compared to 11% drop using the existing ℓ2-approximation. Open code is provided for reproducing and extending our results.