mlsanityData to 14 Oct 2026

ReLoRA: High-Rank Training Through Low-Rank Updates

arXiv 2307.05695 ↗First posted 11 Jul 2023cs.CL cs.LGcited by 204

Not on the map. Why?

Abstract (arXiv)

Despite the dominance and effectiveness of scaling, resulting in large networks with hundreds of billions of parameters, the necessity to train overparameterized models remains poorly understood, while training costs grow exponentially. In this paper, we explore parameter-efficient training techniques as an approach to training large neural networks. We introduce a novel method called ReLoRA, which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to training transformer language models with up to 1.3B parameters and demonstrate comparable performance to regular neural network training. ReLoRA saves up to 5.5Gb of RAM per GPU and improves training speed by 9-40% depending on the model size and hardware setup. Our findings show the potential of parameter-efficient techniques for large-scale pre-training.

Key facts. Cited by 204 papers in the map (data through 14 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

38 of 48 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
26 more
  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2017Mixed Precision Trainingcited by 1,156
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  13. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  15. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  16. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  17. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  18. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  19. 2021Scaling Laws for Neural Machine Translationcited by 111
  20. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
  21. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  22. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 43
  23. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  24. 2020Growing Efficient Deep Networks by Structured Continuous Sparsificationcited by 28
  25. 2023InRank: Incremental Low-Rank Learningcited by 10
  26. 2020HOTCAKE: Higher Order Tucker Articulated Kernels for Deeper CNN Compressioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 60; 2025: 80; 2026: 61.

204 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  2. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  3. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  4. 2024A Survey on LoRA of Large Language Modelscited by 67
  5. 2025The Curse of Depth in Large Language Modelscited by 62
  6. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  7. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  8. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  9. 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 47
  10. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  11. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 43
  12. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
38 more
  1. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  2. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  3. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  4. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  5. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  6. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  7. 2025Tensor Product Attention Is All You Needcited by 31
  8. 2024LoTR: Low Tensor Rank Weight Adaptationcited by 29
  9. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 28
  10. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  11. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  12. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  13. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 22
  14. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 21
  15. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  16. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  17. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  18. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
  19. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 17
  20. 2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shiftscited by 16
  21. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 16
  22. 2024CompAct: Compressed Activations for Memory-Efficient LLM Trainingcited by 15
  23. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  24. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  25. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  26. 2024ZipNN: Lossless Compression for AI Modelscited by 14
  27. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 14
  28. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 14
  29. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13
  30. 2024Searching for Efficient Linear Layers over a Continuous Space of Structured Matricescited by 13
  31. 2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 13
  32. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 13
  33. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  34. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 12
  35. 2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptationcited by 11
  36. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 11
  37. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  38. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 10

50 most cited of 204 All 204 (JSON)