# SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search

arXiv: https://arxiv.org/abs/2208.10404 · HTML: https://mlsanity.com/paper/2208.10404/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 22 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [ZeroQ: A Novel Zero Shot Quantization Framework](https://mlsanity.com/paper/2001.00281/) (2020)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Accelerating Very Deep Convolutional Networks for Classification and Detection](https://mlsanity.com/paper/1505.06798/) (2015)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network](https://mlsanity.com/paper/2008.05441/) (2020)
- [Learning Filter Basis for Convolutional Neural Network Compression](https://mlsanity.com/paper/1908.08932/) (2019)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Search](https://mlsanity.com/paper/1901.11211/) (2019)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Efficient and Accurate Approximations of Nonlinear Convolutional Networks](https://mlsanity.com/paper/1411.4229/) (2014)
- [Post-training deep neural network pruning via layer-wise calibration](https://mlsanity.com/paper/2104.15023/) (2021)
- [Trained Rank Pruning for Efficient Deep Neural Networks](https://mlsanity.com/paper/1812.02402/) (2018)
- [Extreme Network Compression via Filter Group Approximation](https://mlsanity.com/paper/1807.11254/) (2018)
- [Compressing Models with Few Samples: Mimicking then Replacing](https://mlsanity.com/paper/2201.02620/) (2022)
- [Structured Convolutions for Efficient Neural Network Design](https://mlsanity.com/paper/2008.02454/) (2020)
- [Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks](https://mlsanity.com/paper/2006.05066/) (2020)

## Built on by (most cited first, 3 of 3)

- [FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer](https://mlsanity.com/paper/2311.03912/) (2023)
- [BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression](https://mlsanity.com/paper/2509.25136/) (2025)
- [Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor Decomposition](https://mlsanity.com/paper/2306.05021/) (2023)

## Abstract (arXiv)

The task of compressing pre-trained Deep Neural Networks has attracted wide interest of the research community due to its great benefits in freeing practitioners from data access requirements. In this domain, low-rank approximation is a promising method, but existing solutions considered a restricted number of design choices and failed to efficiently explore the design space, which lead to severe accuracy degradation and limited compression ratio achieved. To address the above limitations, this work proposes the SVD-NAS framework that couples the domains of low-rank approximation and neural architecture search. SVD-NAS generalises and expands the design choices of previous works by introducing the Low-Rank architecture space, LR-space, which is a more fine-grained design space of low-rank approximation. Afterwards, this work proposes a gradient-descent-based search for efficiently traversing the LR-space. This finer and more thorough exploration of the possible design choices results in improved accuracy as well as reduction in parameters, FLOPS, and latency of a CNN model. Results demonstrate that the SVD-NAS achieves 2.06-12.85pp higher accuracy on ImageNet than state-of-the-art methods under the data-limited problem setting. SVD-NAS is open-sourced at https://github.com/Yu-Zhewen/SVD-NAS.