# Decomposable-Net: Scalable Low-Rank Compression for Neural Networks

arXiv: https://arxiv.org/abs/1910.13141 · HTML: https://mlsanity.com/paper/1910.13141/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2019.

## Built on by (most cited first, 5 of 5)

- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [Revealing the Utilized Rank of Subspaces of Learning in Neural Networks](https://mlsanity.com/paper/2407.04797/) (2024)
- [STN: Scalable Tensorizing Networks via Structure-Aware Training and Adaptive Compression](https://mlsanity.com/paper/2205.15198/) (2022)
- [A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression](https://mlsanity.com/paper/2309.13077/) (2023)
- [Compression of Recurrent Neural Networks using Matrix Factorization](https://mlsanity.com/paper/2310.12688/) (2023)

## Abstract (arXiv)

Compressing DNNs is important for the real-world applications operating on resource-constrained devices. However, we typically observe drastic performance deterioration when changing model size after training is completed. Therefore, retraining is required to resume the performance of the compressed models suitable for different devices. In this paper, we propose Decomposable-Net (the network decomposable in any size), which allows flexible changes to model size without retraining. We decompose weight matrices in the DNNs via singular value decomposition and adjust ranks according to the target model size. Unlike the existing low-rank compression methods that specialize the model to a fixed size, we propose a novel backpropagation scheme that jointly minimizes losses for both of full- and low-rank networks. This enables not only to maintain the performance of a full-rank network it without retraining but also to improve low-rank networks in multiple sizes. Additionally, we introduce a simple criterion for rank selection that effectively suppresses approximation error. In experiments on the ImageNet classification task, Decomposable-Net yields superior accuracy in a wide range of model sizes. In particular, Decomposable-Net achieves the top-1 accuracy of 73.2% with 0.27×MACs with ResNet-50, compared to Tucker decomposition (67.4% / 0.30×), Trained Rank Pruning (70.6% / 0.28×), and universally slimmable networks (71.4% / 0.26×).