mlsanityData to 14 Oct 2026

Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters

arXiv 2506.14530 ↗First posted 17 Jun 2025stat.ML cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning (PEFT) technique for foundation models. Recent work has highlighted an inherent asymmetry in the initialization of LoRA's low-rank factors, which has been present since its inception and was presumably derived experimentally. This paper focuses on providing a comprehensive theoretical characterization of asymmetric LoRA with frozen random factors. First, while existing research provides upper-bound generalization guarantees based on averages over multiple experiments, the behaviour of a single fine-tuning run with specific random factors remains an open question. We address this by investigating the concentration of the typical LoRA generalization gap around its mean. Our main upper bound reveals a sample complexity of tildeOleft(frac√r√Nright) with high probability for rank r LoRAs trained on N samples. Additionally, we also determine the fundamental limits in terms of sample efficiency, establishing a matching lower bound of Oleft(frac1√Nright). By more closely reflecting the practical scenario of a single fine-tuning run, our findings offer crucial insights into the reliability and practicality of asymmetric LoRA.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

32 of 48 references.

References · 32

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  7. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  9. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  10. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  11. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  12. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
20 more
  1. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  2. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  3. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  4. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  5. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  6. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  7. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 104
  8. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  9. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  10. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  11. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  12. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  13. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  14. 2024Score Distillation via Reparametrized DDIMcited by 41
  15. 2023Efficient Adaptation of Large Vision Transformer via Adapter Re-Composingcited by 35
  16. 2022Towards a Unified View on Visual Parameter-Efficient Transfer Learningcited by 25
  17. 2025Tina: Tiny Reasoning Models via LoRAcited by 25
  18. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  19. 2023PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descentcited by 10
  20. 2022Limitations on approximation by deep and shallow neural networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  2. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  3. 2026LoRA vs. Full Fine-Tuning: A Theoretical Perspectivecited by 2
  4. 2026Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selectioncited by 2
  5. 2025LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMscited by 0
  6. 2026ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equationscited by 0
  7. 2026Training-Free Generative Sampling via Moment-Matched Score Smoothingcited by 0
  8. 2026The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systemscited by 0
  9. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0
  10. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0
  11. 2026Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRAcited by 0