mlsanityData to 7 Oct 2026

Accelerating the Low-Rank Decomposed Models

arXiv 2407.20266 ↗First posted 24 Jul 2024cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Tensor decomposition is a mathematically supported technique for data compression. It consists of applying some kind of a Low Rank Decomposition technique on the tensors or matrices in order to reduce the redundancy of the data. However, it is not a popular technique for compressing the AI models duo to the high number of new layers added to the architecture after decomposition. Although the number of parameters could shrink significantly, it could result in the model be more than twice deeper which could add some latency to the training or inference. In this paper, we present a comprehensive study about how to modify low rank decomposition technique in AI models so that we could benefit from both high accuracy and low memory consumption as well as speeding up the training and inference

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

22 of 35 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  8. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  9. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  10. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  11. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  12. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 118
10 more
  1. 2017Exploring the Regularity of Sparse Structure in Convolutional Neural Networkscited by 37
  2. 2019Fully Quantized Transformer for Machine Translationcited by 36
  3. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  4. 2019A Simplified Fully Quantized Transformer for End-to-end Speech Recognitioncited by 11
  5. 2023GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Valuescited by 9
  6. 2023Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantizationcited by 8
  7. 2023Speeding up Resnet Architecture with Layers Targeted Low Rank Decompositioncited by 8
  8. 2019MUSCO: Multi-Stage Compression of neural networkscited by 5
  9. 2024SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connectioncited by 5
  10. 2024Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Modelcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0