mlsanityData to 7 Oct 2026

ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks

arXiv 2401.10341 ↗First posted 18 Jan 2024cs.CV cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Low-rank compression, a popular model compression technique that produces compact convolutional neural networks (CNNs) with low rankness, has been well-studied in the literature. On the other hand, low-rank training, as an alternative way to train low-rank CNNs from scratch, has been exploited little yet. Unlike low-rank compression, low-rank training does not need pre-trained full-rank models, and the entire training phase is always performed on the low-rank structure, bringing attractive benefits for practical applications. However, the existing low-rank training solutions still face several challenges, such as a considerable accuracy drop and/or still needing to update full-size models during the training. In this paper, we perform a systematic investigation on low-rank CNN training. By identifying the proper low-rank format and performance-improving strategy, we propose ELRT, an efficient low-rank training solution for high-accuracy, high-compactness, low-rank CNN models. Our extensive evaluation results for training various CNNs on different datasets demonstrate the effectiveness of ELRT.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 56 of its 68 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

56 of 68 references.

References · 56

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2015Automatic differentiation in machine learning: a surveycited by 846
  5. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  6. 2013Network In Networkcited by 600
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  8. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  9. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  10. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  11. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  12. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
38 more
  1. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  2. 2015Tensorizing Neural Networkscited by 246
  3. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  4. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  5. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  6. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  7. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  8. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  9. 2020Dynamic Model Pruning with Feedbackcited by 125
  10. 2019Orthogonal Convolutional Neural Networkscited by 122
  11. 2015Convolutional neural networks with low-rank regularizationcited by 102
  12. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  13. 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 92
  14. 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
  15. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
  16. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  17. 2020Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compressioncited by 72
  18. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  19. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  20. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  21. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  22. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  23. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 52
  24. 2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
  25. 2018Wide Compression: Tensor Ring Netscited by 50
  26. 2020DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocationcited by 47
  27. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  28. 2019PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfigurationcited by 41
  29. 2019Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimationcited by 40
  30. 2021Towards Compact CNNs via Collaborative Compressioncited by 36
  31. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  32. 2020TRP: Trained Rank Pruning for Efficient Deep Neural Networkscited by 32
  33. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  34. 2020Growing Efficient Deep Networks by Structured Continuous Sparsificationcited by 28
  35. 2020Heuristic Rank Selection with Progressively Searching Tensor Ring Networkcited by 28
  36. 2015Training CNNs with Low-Rank Filters for Efficient Image Classificationcited by 25
  37. 2019Einconv: Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networkscited by 24
  38. 2019T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensorcited by 22

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  3. 2024LoRTA: Low Rank Tensor Adaptation of Large Language Modelscited by 5
  4. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
  5. 2024Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Compositioncited by 3
  6. 2025HOFT: Householder Orthogonal Fine-tuningcited by 3
  7. 2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
  8. 2026Bayesian Lottery Ticket Hypothesiscited by 2
  9. 2025Efficient CNN Compression via Multi-method Low Rank Factorization and Feature Map Similaritycited by 0
  10. 2026Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retractioncited by 0
  11. 2026Efficient Pre-Training of LLMs through Truncated SVD Representationscited by 0