mlsanityData to 14 Oct 2026

VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks

arXiv 2405.15179 ↗First posted 24 May 2024cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

As the adoption of large language models increases and the need for per-user or per-task model customization grows, the parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. To further reduce stored parameters, we introduce a "divide-and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing parameters globally via a vector bank. As an instantiation of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable top-k admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available at https://github.com/leo-yangli/VB-LoRA. This method has been merged into the Hugging Face PEFT package.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

39 of 65 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
27 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2024Mixtral of Expertscited by 1,823
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  13. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  14. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  15. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  16. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  17. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  18. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  19. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 74
  20. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  21. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  22. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  23. 2022Combining Modular Skills in Multitask Learningcited by 28
  24. 2014Identifiability of the Simplex Volume Minimization Criterion for Blind Hyperspectral Unmixing: The No Pure-Pixel Casecited by 26
  25. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25
  26. 2023Magnitude Invariant Parametrizations Improve Hypernetwork Learningcited by 14
  27. 2023Deep Fusion: Efficient Network Training via Pre-trained Initializationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 21; 2026: 24.

52 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  3. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  4. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 27
  5. 2026Learning to Reason in 13 Parameterscited by 14
  6. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  7. 2024ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptationcited by 11
  8. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  9. 2025QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generationcited by 9
  10. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 9
  11. 2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learningcited by 7
  12. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
38 more
  1. 2025Uni-LoRA: One Vector is All You Needcited by 7
  2. 2025RaSA: Rank-Sharing Low-Rank Adaptationcited by 6
  3. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 6
  4. 2025TARA: Token-Aware LoRA for Composable Personalization in Diffusion Modelscited by 5
  5. 2025WeightLoRA: Keep Only Necessary Adapterscited by 4
  6. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
  7. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  8. 2025BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversitycited by 4
  9. 2025HOFT: Householder Orthogonal Fine-tuningcited by 3
  10. 2025Optimization-Inspired Few-Shot Adaptation for Large Language Modelscited by 3
  11. 2026Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routingcited by 3
  12. 20251LoRA: Summation Compression for Very Low-Rank Adaptationcited by 2
  13. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helpscited by 2
  14. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  15. 2025Meta-LoRA: Meta-Learning LoRA Components for Domain-Aware ID Personalizationcited by 1
  16. 2025TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Modelscited by 1
  17. 2025Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuningcited by 1
  18. 2025A Lightweight Framework for Trigger-Guided LoRA-Based Self-Adaptation in LLMscited by 1
  19. 2026Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languagescited by 1
  20. 2026LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modulescited by 1
  21. 2026LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weightscited by 1
  22. 2026ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuningcited by 1
  23. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  24. 2026Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptationcited by 1
  25. 2025GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Modelscited by 0
  26. 2025Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Servingcited by 0
  27. 2025FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergencecited by 0
  28. 2026ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detectioncited by 0
  29. 2026Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compressioncited by 0
  30. 2026Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atomscited by 0
  31. 2026Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulationcited by 0
  32. 2026Not How Many, But Which: Parameter Placement in Low-Rank Adaptationcited by 0
  33. 2026SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillationcited by 0
  34. 2026Spectral Rank Certification for Foundation Model Adapterscited by 0
  35. 2026Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urducited by 0
  36. 2026Choosing a PEFT Variant for Per-Patient Dysarthric ASR: A Single-Speaker Case Study on Two ASR Basescited by 0
  37. 2026What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptationcited by 0
  38. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  2. 2026Parameter-Efficient Fine-Tuning with Learnable Rank
  3. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  4. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  5. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  6. 2024LoRTA: Low Rank Tensor Adaptation of Large Language Models
  7. 2025LoRA Is Slower Than You Think
  8. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?