mlsanityData to 7 Oct 2026

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

arXiv 2605.31484 ↗First posted 29 May 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

46 of 64 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  9. 2017Learning multiple visual domains with residual adapterscited by 553
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
34 more
  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  2. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  3. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  4. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  5. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  6. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  7. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  8. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  9. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  10. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  11. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  12. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  13. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  14. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  15. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  16. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  17. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  18. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  19. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  20. 2022Beyond the Edge of Stability via Two-step Gradient Updatescited by 38
  21. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 31
  22. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  23. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  24. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  25. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  26. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  27. 2025LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapterscited by 13
  28. 2023Convergence of Alternating Gradient Descent for Matrix Factorizationcited by 12
  29. 2025Riemannian Optimization for LoRA on the Stiefel Manifoldcited by 10
  30. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  31. 2026Path-conditioned training: a principled way to rescale ReLU neural networkscited by 6
  32. 2024Projected Gradient Descent Algorithm for Low-Rank Matrix Estimationcited by 4
  33. 2025Learning Dynamics of Deep Linear Networks Beyond the Edge of Stabilitycited by 4
  34. 2023Gauss-Southwell type descent methods for low-rank matrix optimizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  2. 2026Rotated Manifold Optimization for Low-Rank Adaptationcited by 0