mlsanityData to 7 Oct 2026

VeRA: Vector-based Random Matrix Adaptation

arXiv 2310.11454 ↗First posted 17 Oct 2023cs.CLcited by 294

Not on the map. Why?

Abstract (arXiv)

Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.

Key facts. Cited by 294 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

27 of 40 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
15 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  7. 2021Random Feature Attentioncited by 350
  8. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  9. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  10. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  11. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  12. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  13. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  14. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  15. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 93; 2025: 125; 2026: 73.

294 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  3. 2024LoRA Learns Less and Forgets Lesscited by 313
  4. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  5. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  6. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  7. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  8. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  9. 2024LoFiT: Localized Fine-tuning on LLM Representationscited by 56
  10. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  11. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  12. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
38 more
  1. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  2. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  3. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  4. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  5. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  6. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  7. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  8. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  9. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 30
  10. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  11. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  12. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  13. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  14. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  15. 2025The Universal Weight Subspace Hypothesiscited by 22
  16. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Datacited by 21
  17. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 21
  18. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  19. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  20. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  21. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 18
  22. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  23. 2024Sparse High Rank Adapterscited by 17
  24. 2024PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRAcited by 16
  25. 2024MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuningcited by 16
  26. 2024LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptationcited by 16
  27. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  28. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  29. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  30. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  31. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  32. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  33. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  34. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  35. 2024RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantizationcited by 13
  36. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  37. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  38. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13

50 most cited of 294 All 294 (JSON)