mlsanityData to 7 Oct 2026

Speeding up Resnet Architecture with Layers Targeted Low Rank Decomposition

arXiv 2309.12412 ↗First posted 21 Sep 2023cs.CV cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Compression of a neural network can help in speeding up both the training and the inference of the network. In this research, we study applying compression using low rank decomposition on network layers. Our research demonstrates that to acquire a speed up, the compression methodology should be aware of the underlying hardware as analysis should be done to choose which layers to compress. The advantage of our approach is demonstrated via a case study of compressing ResNet50 and training on full ImageNet-ILSVRC2012. We tested on two different hardware systems Nvidia V100 and Huawei Ascend910. With hardware targeted compression, results on Ascend910 showed 5.36% training speedup and 15.79% inference speed on Ascend310 with only 1% drop in accuracy compared to the original uncompressed model

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

15 of 30 references.

References · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  4. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  5. 2019Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolutioncited by 140
  6. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 118
  7. 2017Exploring the Regularity of Sparse Structure in Convolutional Neural Networkscited by 37
  8. 2019Fully Quantized Transformer for Machine Translationcited by 36
  9. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  10. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  11. 2019A Simplified Fully Quantized Transformer for End-to-end Speech Recognitioncited by 11
  12. 2023Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantizationcited by 8
3 more
  1. 2018Rank Selection of CP-decomposed Convolutional Layers with Variational Bayesian Matrix Factorizationcited by 5
  2. 2018Iterative Low-Rank Approximation for CNN Compressioncited by 2
  3. 2023Improving Resnet-9 Generalization Trained on Small Datasetscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 2; 2026: 2.

8 papers.

  1. 2023GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Valuescited by 9
  2. 2023SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Samplingcited by 3
  3. 2025ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Trainingcited by 3
  4. 2024Accelerating the Low-Rank Decomposed Modelscited by 1
  5. 2024Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?cited by 0
  6. 2025Efficient CNN Compression via Multi-method Low Rank Factorization and Feature Map Similaritycited by 0
  7. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0
  8. 2026Optimizing Tensor Train Decomposition in DNNs for RISC-V Architectures Using Design Space Exploration and Compiler Optimizationscited by 0