mlsanityData to 7 Oct 2026

What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation

arXiv 2609.32002 ↗First posted 25 Sep 2026cs.AI cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The rank r in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets---the idealization of the weight decay and norm control used in practice---this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the matrices of rank at most r inside a nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update---so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every r ≥ 1, its Rademacher complexity carries no dependence on r, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound---though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source--target alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs---not how much capacity the model has.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

32 of 44 references.

References · 32

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  5. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  6. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  7. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  8. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  9. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  10. 2016A vector-contraction inequality for Rademacher complexitiescited by 141
  11. 2016Theoretical Analysis of Domain Adaptation with Optimal Transportcited by 105
  12. 2023The Expressive Power of Low-Rank Adaptationcited by 100
20 more
  1. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  4. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  5. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  6. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 41
  7. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  8. 2024Hyperbolic Fine-Tuning for Large Language Modelscited by 24
  9. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  10. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  11. 2024SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Valuescited by 10
  12. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  13. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  14. 2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
  15. 2025LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  16. 2025Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptationcited by 3
  17. 2026SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuningcited by 2
  18. 2026Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routingcited by 2
  19. 2025Spectral-Aware Low-Rank Adaptation for Speaker Verificationcited by 1
  20. 2026LoRA vs. Full Fine-Tuning: A Theoretical Perspectivecited by 1

Built on by

Not cited yet.