# Learning Low-Rank Approximation for CNNs

arXiv: https://arxiv.org/abs/1905.10145 · HTML: https://mlsanity.com/paper/1905.10145/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 7 of 7)

- [Vision Transformer Pruning](https://mlsanity.com/paper/2104.08500/) (2021)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer](https://mlsanity.com/paper/2311.03912/) (2023)
- [Robust Adversarial Defense by Tensor Factorization](https://mlsanity.com/paper/2309.01077/) (2023)
- [Rotation Invariant Quantization for Model Compression](https://mlsanity.com/paper/2303.03106/) (2023)
- [Vision Transformer Pruning Via Matrix Decomposition](https://mlsanity.com/paper/2308.10839/) (2023)
- [Variational Sparse Paired Autoencoders (vsPAIR) for Inverse Problems and Uncertainty Quantification](https://mlsanity.com/paper/2602.02948/) (2026)

## Abstract (arXiv)

Low-rank approximation is an effective model compression technique to not only reduce parameter storage requirements, but to also reduce computations. For convolutional neural networks (CNNs), however, well-known low-rank approximation methods, such as Tucker or CP decomposition, result in degraded model accuracy because decomposed layers hinder training convergence. In this paper, we propose a new training technique that finds a flat minimum in the view of low-rank approximation without a decomposed structure during training. By preserving the original model structure, 2-dimensional low-rank approximation demanding lowering (such as im2col) is available in our proposed scheme. We show that CNN models can be compressed by low-rank approximation with much higher compression ratio than conventional training methods while maintaining or even enhancing model accuracy. We also discuss various 2-dimensional low-rank approximation techniques for CNNs.