# Speeding up Resnet Architecture with Layers Targeted Low Rank Decomposition

arXiv: https://arxiv.org/abs/2309.12412 · HTML: https://mlsanity.com/paper/2309.12412/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Discrimination-aware Channel Pruning for Deep Neural Networks](https://mlsanity.com/paper/1810.11809/) (2018)
- [Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution](https://mlsanity.com/paper/1904.05049/) (2019)
- [A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1804.03294/) (2018)
- [Exploring the Regularity of Sparse Structure in Convolutional Neural Networks](https://mlsanity.com/paper/1705.08922/) (2017)
- [Fully Quantized Transformer for Machine Translation](https://mlsanity.com/paper/1910.10485/) (2019)
- [MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation](https://mlsanity.com/paper/2105.05912/) (2021)
- [A Short Study on Compressing Decoder-Based Language Models](https://mlsanity.com/paper/2110.08460/) (2021)
- [A Simplified Fully Quantized Transformer for End-to-end Speech Recognition](https://mlsanity.com/paper/1911.03604/) (2019)
- [Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization](https://mlsanity.com/paper/2309.03824/) (2023)
- [Rank Selection of CP-decomposed Convolutional Layers with Variational Bayesian Matrix Factorization](https://mlsanity.com/paper/1801.05243/) (2018)
- [Iterative Low-Rank Approximation for CNN Compression](https://mlsanity.com/paper/1803.08995/) (2018)
- [Improving Resnet-9 Generalization Trained on Small Datasets](https://mlsanity.com/paper/2309.03965/) (2023)

## Built on by (most cited first, 8 of 8)

- [GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values](https://mlsanity.com/paper/2311.03426/) (2023)
- [SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Sampling](https://mlsanity.com/paper/2311.15134/) (2023)
- [ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training](https://mlsanity.com/paper/2505.17331/) (2025)
- [Accelerating the Low-Rank Decomposed Models](https://mlsanity.com/paper/2407.20266/) (2024)
- [Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?](https://mlsanity.com/paper/2407.16514/) (2024)
- [Efficient CNN Compression via Multi-method Low Rank Factorization and Feature Map Similarity](https://mlsanity.com/paper/2510.00062/) (2025)
- [EPAS: Efficient Training with Progressive Activation Sharing](https://mlsanity.com/paper/2601.19089/) (2026)
- [Optimizing Tensor Train Decomposition in DNNs for RISC-V Architectures Using Design Space Exploration and Compiler Optimizations](https://mlsanity.com/paper/2602.01996/) (2026)

## Abstract (arXiv)

Compression of a neural network can help in speeding up both the training and the inference of the network. In this research, we study applying compression using low rank decomposition on network layers. Our research demonstrates that to acquire a speed up, the compression methodology should be aware of the underlying hardware as analysis should be done to choose which layers to compress. The advantage of our approach is demonstrated via a case study of compressing ResNet50 and training on full ImageNet-ILSVRC2012. We tested on two different hardware systems Nvidia V100 and Huawei Ascend910. With hardware targeted compression, results on Ascend910 showed 5.36% training speedup and 15.79% inference speed on Ascend310 with only 1% drop in accuracy compared to the original uncompressed model