mlsanityData to 7 Oct 2026

DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

arXiv 2210.07558 ↗First posted 14 Oct 2022cs.CL cs.LGcited by 229

Not on the map. Why?

Abstract (arXiv)

With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.

Key facts. Cited by 229 papers in the map (data through 7 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

42 of 49 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019Searching for MobileNetV3cited by 1,984
30 more
  1. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  7. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  8. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  9. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  10. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  11. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  12. 2019Structured Pruning of Large Language Modelscited by 201
  13. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  14. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  15. 2014Learning Ordered Representations with Nested Dropoutcited by 81
  16. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  17. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  18. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  19. 2021Annealing Knowledge Distillationcited by 61
  20. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  21. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  22. 2019Fully Quantized Transformer for Machine Translationcited by 36
  23. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  24. 2021Kronecker Decomposition for GPT Compressioncited by 27
  25. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  26. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  27. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16
  28. 2019A Simplified Fully Quantized Transformer for End-to-end Speech Recognitioncited by 11
  29. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  30. 2021From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tablescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 21; 2024: 96; 2025: 69; 2026: 42.

229 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  3. 2023A Survey on Model Compression for Large Language Modelscited by 263
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  5. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  9. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  10. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  11. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 112
  12. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
38 more
  1. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  3. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  4. 2023MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRAcited by 80
  5. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
  6. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 67
  7. 2024A Survey on LoRA of Large Language Modelscited by 67
  8. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  9. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  10. 2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agentcited by 52
  11. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  12. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  13. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  14. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  15. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 41
  16. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  17. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  18. 2024Diffusion-Based Neural Network Weights Generationcited by 38
  19. 2023Visual Tuningcited by 34
  20. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  21. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 31
  22. 2024Continual Forgetting for Pre-trained Vision Modelscited by 29
  23. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
  24. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 26
  25. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25
  26. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  27. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  28. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  29. 2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimizationcited by 19
  30. 2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
  31. 2025Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weightscited by 18
  32. 2023Hyperparameter Optimization for Large Language Model Instruction-Tuningcited by 17
  33. 2025Improved Representation Steering for Language Modelscited by 17
  34. 2024MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuningcited by 16
  35. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  36. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  37. 2024Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruningcited by 15
  38. 2024Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Expertscited by 15

50 most cited of 229 All 229 (JSON)