mlsanityData to 14 Oct 2026

One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

arXiv 2306.07967 ↗First posted 13 Jun 2023cs.LG cs.AI cs.CVcited by 85

Not on the map. Why?

Abstract (arXiv)

We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adapts to new tasks through not only weights but also additional dimensions like activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured vision benchmarks, achieving superior accuracy with fewer parameters and computations. The proposed method on LLaMA-1 and LLaMA-2 also show considerable enhancements compared to the original LoRA in the language domain. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code and models are available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
22 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  3. 2022Visual Prompt Tuningcited by 1,681
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2019Natural Adversarial Examplescited by 1,408
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  10. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  11. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  12. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  13. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  14. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  15. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  16. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  17. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  18. 2022Neural Prompt Searchcited by 135
  19. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  20. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  21. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  22. 2021Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learningcited by 72

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 40; 2025: 19; 2026: 10.

85 papers.

Sort:

By area:

  1. 2023A Survey on Model Compression for Large Language Modelscited by 269
  2. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  3. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  5. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  6. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 104
  7. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  8. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  9. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 74
  10. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  11. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  12. 2024MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learningcited by 44
38 more
  1. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  2. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  3. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  4. 2024Visual Fourier Prompt Tuningcited by 30
  5. 2024Continual Forgetting for Pre-trained Vision Modelscited by 29
  6. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  7. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  8. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  9. 2024MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Interventioncited by 20
  10. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  11. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  12. 2025DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformerscited by 17
  13. 2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shiftscited by 16
  14. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  15. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
  16. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  17. 2023Diversified in-domain synthesis with efficient fine-tuning for few-shot classificationcited by 11
  18. 2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptationcited by 11
  19. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 11
  20. 2023Initialization Matters for Adversarial Transfer Learningcited by 10
  21. 2024IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Modelscited by 10
  22. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  23. 2023AutoLoRa: A Parameter-Free Automated Robust Fine-Tuning Frameworkcited by 9
  24. 2024Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generationcited by 8
  25. 2023Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Expertscited by 7
  26. 2024Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Datacited by 7
  27. 2024AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Modelscited by 7
  28. 2024PETALface: Parameter Efficient Transfer Learning for Low-resolution Face Recognitioncited by 7
  29. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  30. 2023PILLOW: Enhancing Efficient Instruction Fine-tuning via Prompt Matchingcited by 6
  31. 2024MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuningcited by 6
  32. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  33. 2025MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidancecited by 6
  34. 2024LORS: Low-rank Residual Structure for Parameter-Efficient Network Stackingcited by 5
  35. 2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningcited by 5
  36. 2024MINI-LLM: Memory-Efficient Structured Pruning for Large Language Modelscited by 5
  37. 2025TARA: Token-Aware LoRA for Composable Personalization in Diffusion Modelscited by 5
  38. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a surveycited by 5

50 most cited of 85 All 85 (JSON)