# Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization

arXiv: https://arxiv.org/abs/2507.01841 · HTML: https://mlsanity.com/paper/2507.01841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 59 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Builds on (most cited first, 30 of 37)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition](https://mlsanity.com/paper/1503.02101/) (2015)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [Neural Prompt Search](https://mlsanity.com/paper/2206.04673/) (2022)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [KronA: Parameter Efficient Tuning with Kronecker Adapter](https://mlsanity.com/paper/2212.10650/) (2022)
- [Matrix Completion has No Spurious Local Minimum](https://mlsanity.com/paper/1605.07272/) (2016)
- [A Kernel-Based View of Language Model Fine-Tuning](https://mlsanity.com/paper/2210.05643/) (2022)
- [Bayesian Low-rank Adaptation for Large Language Models](https://mlsanity.com/paper/2308.13111/) (2023)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)
- [The Expressive Power of Low-Rank Adaptation](https://mlsanity.com/paper/2310.17513/) (2023)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters](https://mlsanity.com/paper/2405.17604/) (2024)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors](https://mlsanity.com/paper/2405.19597/) (2024)
- [BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models](https://mlsanity.com/paper/2406.11675/) (2024)
- [LoRA Training in the NTK Regime has No Spurious Local Minima](https://mlsanity.com/paper/2402.11867/) (2024)
- [Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies](https://mlsanity.com/paper/2410.19878/) (2024)
- [ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models](https://mlsanity.com/paper/2403.16187/) (2024)
- [Guarantees for Greedy Maximization of Non-submodular Functions with Applications](https://mlsanity.com/paper/1703.02100/) (2017)
- [LoTR: Low Tensor Rank Weight Adaptation](https://mlsanity.com/paper/2402.01376/) (2024)
- [Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs](https://mlsanity.com/paper/2110.10545/) (2021)
- [The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.03864/) (2024)
- [LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)](https://mlsanity.com/paper/2502.09376/) (2025)
- [PoLAR: Polar-Decomposed Low-Rank Adapter Representation](https://mlsanity.com/paper/2506.03133/) (2025)
- [Personalized LoRA for Human-Centered Text Understanding](https://mlsanity.com/paper/2403.06208/) (2024)

## Abstract (arXiv)

In this paper, we propose SubLoRA, a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function maximization. In contrast to prior approaches, such as AdaLoRA, that rely on first-order (linearized) approximations of the loss function, SubLoRA utilizes second-order information to capture the potentially complex loss landscape by incorporating the Hessian matrix. We show that the linearization becomes inaccurate and ill-conditioned when the LoRA parameters have been well optimized, motivating the need for a more reliable and nuanced second-order formulation. To this end, we reformulate the rank determination problem as a combinatorial optimization problem with a quadratic objective. However, solving this problem exactly is NP-hard in general. To overcome the computational challenge, we introduce a submodular function maximization framework and devise a greedy algorithm with approximation guarantees. We derive a sufficient and necessary condition under which the rank-determination objective becomes submodular, and construct a closed-form projection of the Hessian matrix that satisfies this condition while maintaining computational efficiency. Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency. We further extend SubLoRA to a joint optimization setting, alternating between LoRA parameter updates and rank determination under a rank budget constraint. Extensive experiments on fine-tuning physics-informed neural networks (PINNs) for solving partial differential equations (PDEs) demonstrate the effectiveness of our approach. Results show that SubLoRA outperforms existing methods in both rank determination and joint training performance.