mlsanityData to 7 Oct 2026

Batched Low-Rank Adaptation of Foundation Models

arXiv 2312.05677 ↗First posted 9 Dec 2023cs.LG cs.AI cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) has recently gained attention for fine-tuning foundation models by incorporating trainable low-rank matrices, thereby reducing the number of trainable parameters. While LoRA offers numerous advantages, its applicability for real-time serving to a diverse and global user base is constrained by its incapability to handle multiple task-specific adapters efficiently. This imposes a performance bottleneck in scenarios requiring personalized, task-specific adaptations for each incoming request. To mitigate this constraint, we introduce Fast LoRA (FLoRA), a framework in which each input example in a minibatch can be associated with its unique low-rank adaptation weights, allowing for efficient batching of heterogeneous requests. We empirically demonstrate that FLoRA retains the performance merits of LoRA, showcasing competitive results on the MultiPL-E code generation benchmark spanning over 8 languages and a multilingual speech recognition task across 6 languages.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 9 Dec 2023.

Builds on

45 of 58 references.

References · 45

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
33 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  15. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  16. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  17. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  18. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  19. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  20. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  21. 2023From Sparse to Soft Mixtures of Expertscited by 245
  22. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  23. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  24. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  25. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  26. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  27. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  28. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  29. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  30. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  31. 2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38
  32. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  33. 2023Measuring The Impact Of Programming Language Distributioncited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 10; 2026: 6.

32 papers.

Sort:

By area:

  1. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  2. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  3. 2024A Survey on LoRA of Large Language Modelscited by 67
  4. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 57
  5. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  6. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  7. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  8. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  9. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  10. 20243-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composabilitycited by 10
  11. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  12. 2024Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptationcited by 8
20 more
  1. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  2. 2024Low-Rank Knowledge Decomposition for Medical Foundation Modelscited by 5
  3. 2024LoRTA: Low Rank Tensor Adaptation of Large Language Modelscited by 5
  4. 2024LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Modelscited by 4
  5. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  6. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  7. 2025Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Modelscited by 3
  8. 2026High-Rank Structured Modulation for Parameter-Efficient Fine-Tuningcited by 2
  9. 2026Shared LoRA Subspaces for almost Strict Continual Learningcited by 2
  10. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  11. 2026SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuningcited by 2
  12. 2025CBP-Tuning: Efficient Local Customization for Black-box Large Language Modelscited by 1
  13. 2025FOAM: Blocked State Folding for Memory-Efficient LLM Trainingcited by 1
  14. 2024Synthesize, Partition, then Adapt: Eliciting Diverse Samples from Foundation Modelscited by 0
  15. 2025RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Modelscited by 0
  16. 2025MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Expertscited by 0
  17. 2025Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challengescited by 0
  18. 2025MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Promptscited by 0
  19. 2026Low-Rank Ternary Adaptation for Fine-Tuning Transformerscited by 0
  20. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0