# TRP: Trained Rank Pruning for Efficient Deep Neural Networks

arXiv: https://arxiv.org/abs/2004.14566 · HTML: https://mlsanity.com/paper/2004.14566/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 32)

- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Compression-aware Training of Neural Networks using Frank-Wolfe](https://mlsanity.com/paper/2205.11921/) (2022)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks](https://mlsanity.com/paper/2401.10341/) (2024)
- [Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression](https://mlsanity.com/paper/2104.12416/) (2021)
- [PELA: Learning Parameter-Efficient Models with Low-Rank Approximation](https://mlsanity.com/paper/2310.10700/) (2023)
- [Toward Compact Deep Neural Networks via Energy-Aware Pruning](https://mlsanity.com/paper/2103.10858/) (2021)
- [Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis](https://mlsanity.com/paper/2407.09835/) (2024)
- [Model compression as constrained optimization, with application to neural nets. Part V: combining compressions](https://mlsanity.com/paper/2107.04380/) (2021)
- [Towards Memory-Efficient Neural Networks via Multi-Level in situ Generation](https://mlsanity.com/paper/2108.11430/) (2021)
- [Compact Model Training by Low-Rank Projection with Energy Transfer](https://mlsanity.com/paper/2204.05566/) (2022)
- [FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer](https://mlsanity.com/paper/2311.03912/) (2023)
- [Federated Learning in IoT: a Survey from a Resource-Constrained Perspective](https://mlsanity.com/paper/2308.13157/) (2023)
- [Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control](https://mlsanity.com/paper/2310.03915/) (2023)
- [Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers](https://mlsanity.com/paper/2406.16450/) (2024)
- [Low-Rank Compression for IMC Arrays](https://mlsanity.com/paper/2502.07820/) (2025)
- [RGP: Neural Network Pruning through Its Regular Graph Structure](https://mlsanity.com/paper/2110.15192/) (2021)
- [SNF: Filter Pruning via Searching the Proper Number of Filters](https://mlsanity.com/paper/2112.07282/) (2021)
- [STN: Scalable Tensorizing Networks via Structure-Aware Training and Adaptive Compression](https://mlsanity.com/paper/2205.15198/) (2022)
- [Layer-adaptive Structured Pruning Guided by Latency](https://mlsanity.com/paper/2305.14403/) (2023)
- [A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression](https://mlsanity.com/paper/2309.13077/) (2023)
- [Dynamic Shuffle: An Efficient Channel Mixture Method](https://mlsanity.com/paper/2310.02776/) (2023)
- [Compression of Recurrent Neural Networks using Matrix Factorization](https://mlsanity.com/paper/2310.12688/) (2023)
- [Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training](https://mlsanity.com/paper/2511.04485/) (2025)
- [SLORR: Simple and Efficient In-Training Low-Rank Regularization](https://mlsanity.com/paper/2607.08754/) (2026)
- [Compressing Deep CNNs using Basis Representation and Spectral Fine-tuning](https://mlsanity.com/paper/2105.10436/) (2021)
- [Survey on Computer Vision Techniques for Internet-of-Things Devices](https://mlsanity.com/paper/2308.02553/) (2023)
- [Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning](https://mlsanity.com/paper/2401.08014/) (2024)

## Abstract (arXiv)

To enable DNNs on edge devices like mobile phones, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations. Several previous works attempted to directly approximate a pretrained model by low-rank decomposition; however, small approximation errors in parameters can ripple over a large prediction loss. As a result, performance usually drops significantly and a sophisticated effort on fine-tuning is required to recover accuracy. Apparently, it is not optimal to separate low-rank approximation from training. Unlike previous works, this paper integrates low rank approximation and regularization into the training process. We propose Trained Rank Pruning (TRP), which alternates between low rank approximation and training. TRP maintains the capacity of the original network while imposing low-rank constraints during training. A nuclear regularization optimized by stochastic sub-gradient descent is utilized to further promote low rank in TRP. The TRP trained network inherently has a low-rank structure, and is approximated with negligible performance loss, thus eliminating the fine-tuning process after low rank decomposition. The proposed method is comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous compression methods using low rank approximation.