mlsanityData to 7 Oct 2026

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

arXiv 2311.11501 ↗First posted 20 Nov 2023cs.LG cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

LoRA achieves remarkable resource efficiency and comparable performance when adapting LLMs for specific tasks. Since ChatGPT demonstrated superior performance on various tasks, there has been a growing desire to adapt one model for all tasks. However, the explicit low-rank of LoRA limits the adaptation performance in complex multi-task scenarios. LoRA is dominated by a small number of top singular vectors while fine-tuning decomposes into a set of less important unitary transforms. In this paper, we propose MultiLoRA for better multi-task adaptation by reducing the dominance of top singular vectors observed in LoRA. MultiLoRA scales LoRA modules horizontally and change parameter initialization of adaptation matrices to reduce parameter dependency, thus yields more balanced unitary subspaces. We unprecedentedly construct specialized training data by mixing datasets of instruction follow, natural language understanding, world knowledge, to cover semantically and syntactically different samples. With only 2.5% of additional parameters, MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple benchmarks and model scales. Further investigation into weight update matrices of MultiLoRA exhibits reduced dependency on top singular vectors and more democratic unitary transform contributions.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

27 of 32 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2022Locating and Editing Factual Associations in GPTcited by 2,609
15 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2016An overview of gradient descent optimization algorithmscited by 1,021
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  6. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  9. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  10. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  11. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  12. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  13. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  14. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  15. 2023One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarizationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 15; 2026: 5.

32 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  3. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  4. 2024A Survey on LoRA of Large Language Modelscited by 67
  5. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 37
  6. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  7. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  8. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  9. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  10. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 13
  11. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  12. 2025Ensembles of Low-Rank Expert Adapterscited by 9
20 more
  1. 2024Task-Aware Low-Rank Adaptation of Segment Anything Modelcited by 6
  2. 2024Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Modelcited by 6
  3. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
  4. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 4
  5. 2024Multi LoRA Meets Vision: Merging multiple adapters to create a multi task modelcited by 3
  6. 2025Pay Attention to Small Weightscited by 3
  7. 2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
  8. 2024SBoRA: Low-Rank Adaptation with Regional Weight Updatescited by 2
  9. 2025MetaLoRA: Tensor-Enhanced Adaptive Low-Rank Fine-tuningcited by 2
  10. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  11. 2025KARST: Multi-Kernel Kronecker Adaptation with Re-Scaling Transmission for Visual Classificationcited by 1
  12. 2025Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradientscited by 1
  13. 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
  14. 2025MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptationcited by 1
  15. 2025MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Modelscited by 1
  16. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  17. 2026ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuningcited by 1
  18. 2026Crab+: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 0
  19. 2026PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Promptscited by 0
  20. 2026The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Predictioncited by 0