mlsanityData to 7 Oct 2026

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

arXiv 2408.10280 ↗First posted 18 Aug 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Nested Low-Rank Adaptation (NoRA), a novel approach to parameter-efficient fine-tuning that extends the capabilities of Low-Rank Adaptation (LoRA) techniques. Vanilla LoRA overlooks pre-trained weight inheritance and still requires fine-tuning numerous parameters. To addresses these issues, our NoRA adopts a dual-layer nested structure with Singular Value Decomposition (SVD), effectively leveraging original matrix knowledge while reducing tunable parameters. Specifically, NoRA freezes the outer LoRA weights and utilizes an inner LoRA design, providing enhanced control over model optimization. This approach allows the model to more precisely adapt to specific tasks while maintaining a compact parameter space. By freezing outer LoRA weights and using an inner LoRA design, NoRA enables precise task adaptation with a compact parameter space. Evaluations on tasks including commonsense reasoning with large language models, fine-tuning vision-language models, and subject-driven generation demonstrate NoRA's superiority over LoRA and its variants. Code will be released upon acceptance.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

47 of 58 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
35 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  6. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  9. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  10. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  11. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  12. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  13. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  14. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  15. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  16. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  17. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  18. 2022Task Residual for Tuning Vision-Language Modelscited by 122
  19. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  20. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  21. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  22. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  23. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  24. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  25. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  26. 2024A Survey on LoRA of Large Language Modelscited by 67
  27. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  28. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  29. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  30. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  31. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  32. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  33. 2023EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantizationcited by 22
  34. 2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Modelscited by 14
  35. 2023Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discoverycited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  2. 2026Learning to Reason in 13 Parameterscited by 13
  3. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  4. 2025DenseLoRA: Dense Low-Rank Adaptation of Large Language Modelscited by 4
  5. 2024FineGates: LLMs Finetuning with Compression using Stochastic Gatescited by 3
  6. 2025WeightLoRA: Keep Only Necessary Adapterscited by 3
  7. 2025LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Modelcited by 2
  8. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  9. 2025Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptationcited by 1
  10. 2025GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptationcited by 1
  11. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 1
  12. 2026TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Modelscited by 0
1 more
  1. 2026LoRA Enhanced Contrastive Learning with SAS Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  2. 2026Parameter-Efficient Fine-Tuning with Learnable Rank
  3. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  4. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation
  5. 2025DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  6. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  7. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  8. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning