mlsanityData to 13 Oct 2026

ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation

arXiv 2406.10785 ↗First posted 16 Jun 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Shared Low Rank Adaptation (ShareLoRA), a Large Language Model (LLM) fine-tuning technique that balances parameter efficiency, adaptability, and robustness without compromising performance. By strategically sharing the low-rank weight matrices across different layers, ShareLoRA achieves 44% to 96% reduction in trainable parameters compared to standard LoRA, alongside a substantial decrease in memory overhead. This efficiency gain scales with model size, making ShareLoRA particularly advantageous for resource-constrained environments. Importantly, ShareLoRA not only maintains model performance but also exhibits robustness in both classification and generation tasks across diverse models, including RoBERTa, GPT-2, and LLaMA series (1, 2, and 3). It consistently outperforms LoRA in zero-shot, few-shot, and continual fine-tuning scenarios, achieving up to 1.2% average accuracy improvement, and enhanced generalization across domains. In continual learning settings, ShareLoRA achieves 1.2% higher accuracy on GSM8K, 0.6% on HumanEval, and 0.5% on both MMLU and MMLU-Pro. Our results demonstrate that ShareLoRA supports high-quality fine-tuning while offering strong generalization and continual adaptation across various model scales and diverse tasks.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

37 of 49 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  11. 2020Scaling Laws for Neural Language Modelscited by 6,494
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
25 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,240
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 891
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 868
  13. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 362
  14. 2023VeRA: Vector-based Random Matrix Adaptationcited by 297
  15. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 281
  16. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  17. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  18. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  19. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  20. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 134
  21. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  22. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  23. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  24. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  25. 2023MerA: Merging Pretrained Adapters For Few-Shot Learningcited by 13

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 3.

11 papers.

  1. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  2. 2026Learning to Reason in 13 Parameterscited by 13
  3. 2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
  4. 2025PiCa: Parameter-Efficient Fine-Tuning with Column Space Projectioncited by 6
  5. 2025LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  6. 2024ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layerscited by 3
  7. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  8. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  9. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  10. 2025WaRA: Wavelet Low-Rank Adaptation for Medical Image Classificationcited by 1
  11. 2026Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atomscited by 0