mlsanityData to 14 Oct 2026

WeightLoRA: Keep Only Necessary Adapters

arXiv 2506.02724 ↗First posted 3 Jun 2025cs.LG math.OCcited by 4

Not on the map. Why?

Abstract (arXiv)

The widespread utilization of language models in modern applications is inconceivable without Parameter-Efficient Fine-Tuning techniques, such as low-rank adaptation (textttLoRA), which adds trainable adapters to selected layers. Although textttLoRA may obtain accurate solutions, it requires significant memory to train large models and intuition on which layers to add adapters. In this paper, we propose a novel method, textttWeightLoRA, which overcomes this issue by adaptive selection of the most critical textttLoRA heads throughout the optimization process. As a result, we can significantly reduce the number of trainable parameters while maintaining the capability to obtain consistent or even superior metric values. We conduct experiments for a series of competitive benchmarks and DeBERTa, BART, and Llama models, comparing our method with different adaptive approaches. The experimental results demonstrate the efficacy of textttWeightLoRA and the superior performance of textttWeightLoRA+ in almost all cases.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

31 of 39 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
19 more
  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  3. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  4. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  5. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  6. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  7. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  8. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  9. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 43
  10. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  11. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  12. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  13. 2016A Tight Bound of Hard Thresholdingcited by 16
  14. 2014Linear Convergence of Stochastic Iterative Greedy Algorithms with Sparse Constraintscited by 13
  15. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  16. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  17. 2024The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Informationcited by 8
  18. 2022Stability and Risk Bounds of Iterative Hard Thresholdingcited by 7
  19. 2022Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivitycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Learning to Reason in 13 Parameterscited by 14
  2. 2025DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioningcited by 1
  3. 2026LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updatescited by 0
  4. 2026ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuningcited by 0