mlsanityData to 7 Oct 2026

Flora: Low-Rank Adapters Are Secretly Gradient Compressors

arXiv 2402.03293 ↗First posted 5 Feb 2024cs.LG cs.AI stat.MLcited by 103

Not on the map. Why?

Abstract (arXiv)

Despite large neural networks demonstrating remarkable abilities to complete different tasks, they require excessive memory usage to store the optimization states for training. To alleviate this, the low-rank adaptation (LoRA) is proposed to reduce the optimization states by training fewer parameters. However, LoRA restricts overall weight update matrices to be low-rank, limiting the model performance. In this work, we investigate the dynamics of LoRA and identify that it can be approximated by a random projection. Based on this observation, we propose Flora, which is able to achieve high-rank updates by resampling the projection matrices while enjoying the sublinear space complexity of optimization states. We conduct experiments across different tasks and model architectures to verify the effectiveness of our approach.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

32 of 55 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
20 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  6. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  7. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  8. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  9. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  11. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
  12. 2014A Tutorial on Principal Component Analysiscited by 139
  13. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  14. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  15. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  16. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  17. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  18. 2013Experiments with Random Projectioncited by 27
  19. 2020Randomized Automatic Differentiationcited by 23
  20. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 47; 2026: 24.

103 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  3. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  4. 2024A Survey on LoRA of Large Language Modelscited by 67
  5. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  6. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  7. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  8. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  9. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  10. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  11. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  12. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
38 more
  1. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  2. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  3. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 19
  4. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  5. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  6. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 17
  7. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  8. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  9. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  10. 2024CompAct: Compressed Activations for Memory-Efficient LLM Trainingcited by 15
  11. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  12. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 14
  13. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  14. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  15. 2024NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networkscited by 12
  16. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  17. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 11
  18. 2024Low-Rank Interconnected Adaptation across Layerscited by 10
  19. 2024DeMo: Decoupled Momentum Optimizationcited by 10
  20. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 10
  21. 2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
  22. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  23. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 9
  24. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  25. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  26. 2024COAP: Memory-Efficient Training with Correlation-Aware Gradient Projectioncited by 8
  27. 2024GaLore+: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projectioncited by 8
  28. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  29. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  30. 2024AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Modelscited by 7
  31. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 7
  32. 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
  33. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 6
  34. 2024PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuningcited by 5
  35. 2025CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Modelscited by 5
  36. 2025TokMem: One-Token Procedural Memory for Large Language Modelscited by 5
  37. 2025LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  38. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 4

50 most cited of 103 All 103 (JSON)