mlsanityData to 14 Oct 2026

GaLore+: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection

arXiv 2412.19820 ↗First posted 15 Dec 2024cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent low-rank training methods, such as GaLore, have significantly reduced the memory required to optimize large language models (LLMs). However, these methods often suffer from time-consuming low-rank projection estimations. In particular, the singular value decomposition (SVD) in GaLore can consume more than 80% of the total training time. To address this issue, we propose GaLore+, which uses cross-head low-rank projection to reduce the substantial time consumption in estimating low-rank projections for multi-head attention. In addition, we employ randomized subspace iteration to achieve fast SVD. To further enhance performance, we propose sparsely coded residuals to reduce the errors caused by low-rank approximation on the first- and second-order moments of the optimizers and weight updates. We evaluate GaLore+ on arithmetic reasoning and natural language generation datasets. Our experiments demonstrate that GaLore+ delivers superior performance while achieving approximately 4× fine-tuning speed compared to vanilla GaLore.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

34 of 42 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
22 more
  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  2. 2017Learning multiple visual domains with residual adapterscited by 557
  3. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  4. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  5. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  6. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  7. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  8. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  9. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  10. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  11. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  12. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  13. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  14. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  15. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  16. 2020Multi-Head Attention: Collaborate Instead of Concatenatecited by 60
  17. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  18. 2016Fast Algorithms for Robust PCA via Gradient Descentcited by 39
  19. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 23
  20. 2023INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptationcited by 20
  21. 2017Robust PCA by Manifold Optimizationcited by 13
  22. 2014A variational approach to stable principal component pursuitcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  2. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 13
  3. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 9
  4. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 9
  5. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 5
  6. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 5
  7. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 5
  8. 2025Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
  2. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
  3. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  4. 2025OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation
  5. 2024DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
  6. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
  7. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
  8. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation