mlsanityData to 7 Oct 2026

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

arXiv 2401.11316 ↗First posted 20 Jan 2024cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

With the proliferation of large pre-trained language models (PLMs), fine-tuning all model parameters becomes increasingly inefficient, particularly when dealing with numerous downstream tasks that entail substantial training and storage costs. Several approaches aimed at achieving parameter-efficient fine-tuning (PEFT) have been proposed. Among them, Low-Rank Adaptation (LoRA) stands out as an archetypal method, incorporating trainable rank decomposition matrices into each target module. Nevertheless, LoRA does not consider the varying importance of each layer. To address these challenges, we introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process, considering both the temporary magnitude of weights and the accumulated statistics of the input to any given layer. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 39 of its 44 references that are in the map. First posted on arXiv 20 Jan 2024.

Builds on

39 of 44 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2016Layer Normalizationcited by 5,579
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
27 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  12. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  13. 2018Rethinking the Value of Network Pruningcited by 595
  14. 2017Learning multiple visual domains with residual adapterscited by 553
  15. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  16. 2019The State of Sparsity in Deep Neural Networkscited by 471
  17. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  18. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  19. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  20. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  21. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  22. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  23. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  24. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  25. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  26. 2022MAEDAY: MAE for few and zero shot AnomalY-Detectioncited by 30
  27. 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 2.

8 papers.

  1. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  2. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  3. 2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
  4. 2024LoRTA: Low Rank Tensor Adaptation of Large Language Modelscited by 5
  5. 2026IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoringcited by 5
  6. 2024PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuningcited by 0
  7. 2025The Quest for Winning Tickets in Low-Rank Adapterscited by 0
  8. 2026Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimationcited by 0