mlsanityData to 14 Oct 2026

Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization

arXiv 2507.01841 ↗First posted 2 Jul 2025cs.LG cs.IT eess.SPcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose SubLoRA, a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function maximization. In contrast to prior approaches, such as AdaLoRA, that rely on first-order (linearized) approximations of the loss function, SubLoRA utilizes second-order information to capture the potentially complex loss landscape by incorporating the Hessian matrix. We show that the linearization becomes inaccurate and ill-conditioned when the LoRA parameters have been well optimized, motivating the need for a more reliable and nuanced second-order formulation. To this end, we reformulate the rank determination problem as a combinatorial optimization problem with a quadratic objective. However, solving this problem exactly is NP-hard in general. To overcome the computational challenge, we introduce a submodular function maximization framework and devise a greedy algorithm with approximation guarantees. We derive a sufficient and necessary condition under which the rank-determination objective becomes submodular, and construct a closed-form projection of the Hessian matrix that satisfies this condition while maintaining computational efficiency. Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency. We further extend SubLoRA to a joint optimization setting, alternating between LoRA parameter updates and rank determination under a rank budget constraint. Extensive experiments on fine-tuning physics-informed neural networks (PINNs) for solving partial differential equations (PDEs) demonstrate the effectiveness of our approach. Results show that SubLoRA outperforms existing methods in both rank determination and joint training performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

37 of 59 references.

References · 37

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Visual Prompt Tuningcited by 1,681
  3. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  5. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  6. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  7. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  8. 2022Neural Prompt Searchcited by 135
  9. 2024Implicit Style-Content Separation using B-LoRAcited by 135
  10. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  11. 2016Matrix Completion has No Spurious Local Minimumcited by 125
  12. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
25 more
  1. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  2. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  3. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  4. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  6. 2024A Survey on LoRA of Large Language Modelscited by 67
  7. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  8. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  9. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  10. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  11. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  12. 2017Guarantees for Greedy Maximization of Non-submodular Functions with Applicationscited by 35
  13. 2024LoTR: Low Tensor Rank Weight Adaptationcited by 29
  14. 2021Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubscited by 28
  15. 2024The Challenges of the Nonlinear Regime for Physics-Informed Neural Networkscited by 21
  16. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
  17. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  18. 2024Personalized LoRA for Human-Centered Text Understandingcited by 17
  19. 2025Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptationcited by 15
  20. 2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Modelscited by 12
  21. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  22. 2018Tensor Decomposition for Compressing Recurrent Neural Networkcited by 10
  23. 2024LoRA-C: Parameter-Efficient Fine-Tuning of Robust CNN for IoT Devicescited by 10
  24. 2023HyperLoRA for PDEscited by 4
  25. 2025Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networkscited by 1

Built on by

Not cited yet.