mlsanityData to 7 Oct 2026

Exploring Low Rank Training of Deep Neural Networks

arXiv 2209.13569 ↗First posted 27 Sep 2022cs.LG stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks in low rank, i.e. with factorised layers, is of particular interest to the community: it offers efficiency over unfactorised training in terms of both memory consumption and training time. Prior work has focused on low rank approximations of pre-trained networks and training in low rank space with additional objectives, offering various ad hoc explanations for chosen practice. We analyse techniques that work well in practice, and through extensive ablations on models such as GPT2 we provide evidence falsifying common beliefs in the field, hinting in the process at exciting research opportunities that still need answering.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

19 of 22 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2016Wide Residual Networkscited by 3,057
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  10. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  11. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  12. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
7 more
  1. 2015Convolutional neural networks with low-rank regularizationcited by 102
  2. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  3. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  4. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  5. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  6. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  7. 2019Stabilizing the Lottery Ticket Hypothesiscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 10; 2026: 5.

30 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  3. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  4. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  5. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  6. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  7. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  8. 2023The Expressive Power of Tuning Only the Normalization Layerscited by 13
  9. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  10. 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
  11. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  12. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
18 more
  1. 2025LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossingcited by 6
  2. 2024BSDA: Bayesian Random Semantic Data Augmentation for Medical Image Classificationcited by 5
  3. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 4
  4. 2024Structured and Balanced Multi-Component and Multi-Layer Neural Networkscited by 3
  5. 2023Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detectioncited by 2
  6. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  7. 2026Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximationcited by 2
  8. 2023Flat Minima in Linear Estimation and an Extended Gauss Markov Theoremcited by 1
  9. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  10. 2025Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Trainingcited by 1
  11. 2026DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Trainingcited by 1
  12. 2026SLORR: Simple and Efficient In-Training Low-Rank Regularizationcited by 1
  13. 2023Point Cloud Network: An Order of Magnitude Improvement in Linear Layer Parameter Countcited by 0
  14. 2024On Sampling Strategies for Spectral Model Shardingcited by 0
  15. 2025A Low-complexity Structured Neural Network Approach to Intelligently Realize Wideband Multi-beam Beamformerscited by 0
  16. 2025Accelerating Attention with Basis Decompositioncited by 0
  17. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsitycited by 0
  18. 2026Efficient Pre-Training of LLMs through Truncated SVD Representationscited by 0