On the Duality between Gradient Transformations and Adapters
Not on the map. Why?
Abstract (arXiv)
We study memory-efficient optimization of neural networks (in particular language models) with linear gradient transformations, where the gradients are linearly mapped to a lower dimensional space than the full parameter space, thus saving memory required for gradient accumulation and optimizer state persistence. The model parameters are updated by first performing an optimization step in the lower dimensional space and then going back into the original parameter space via the linear map's transpose. We show that optimizing the model in this transformed space is equivalent to reparameterizing the original model through a linear adapter that additively modifies the model parameters, and then only optimizing the adapter's parameters. When the transformation is Kronecker-factored, this establishes an equivalence between GaLore and one-sided LoRA. We show that this duality between gradient transformations and adapter-based reparameterizations unifies existing approaches to memory-efficient training and suggests new techniques for improving training efficiency and memory use.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 53 of its 60 references that are in the map. First posted on arXiv 19 Feb 2025.
Builds on
53 of 60 references.
References · 53
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
- 2020Adaptive Federated Optimizationcited by 1,046
- 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
- 2016Training Deep Nets with Sublinear Memory Costcited by 848
- 2016Learning to learn by gradient descent by gradient descentcited by 620
38 more
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
- 20218-bit Optimizers via Block-wise Quantizationcited by 403
- 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
- 2023Fine-Tuning Language Models with Just Forward Passescited by 345
- 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
- 2021Training Neural Networks with Fixed Sparse Maskscited by 214
- 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
- 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
- 2016Learning to Optimizecited by 195
- 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
- 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
- 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
- 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
- 2017Learned Optimizers that Scale and Generalizecited by 110
- 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
- 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
- 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
- 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
- 2017Neural Optimizer Search with Reinforcement Learningcited by 82
- 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
- 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
- 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
- 2023Memory Efficient Optimizers with 4-bit Statescited by 54
- 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
- 2023Randomized Numerical Linear Algebra : A Perspective on the Field With an Eye to Softwarecited by 38
- 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
- 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
- 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
- 2024OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Trainingcited by 24
- 2020Randomized Automatic Differentiationcited by 23
- 2020Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Networkcited by 20
- 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
- 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
- 2018Faster Neural Network Training with Approximate Tensor Operationscited by 14
- 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
- 2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptationcited by 11
- 2023InRank: Incremental Low-Rank Learningcited by 10
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.