mlsanityData to 7 Oct 2026

Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning

arXiv 2406.16989 ↗First posted 24 Jun 2024cs.LG cs.AIcited by 14

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Low-Rank Adaptation (LoRA) offers an efficient way to fine-tune large language models (LLMs). Its modular and plug-and-play nature allows the integration of various domain-specific LoRAs, enhancing LLM capabilities. Open-source platforms like Huggingface and Modelscope have introduced a new computational paradigm, Uploadable Machine Learning (UML). In UML, contributors use decentralized data to train specialized adapters, which are then uploaded to a central platform to improve LLMs. This platform uses these domain-specific adapters to handle mixed-task requests requiring personalized service. Previous research on LoRA composition either focuses on specific tasks or fixes the LoRA selection during training. However, in UML, the pool of LoRAs is dynamically updated with new uploads, requiring a generalizable selection mechanism for unseen LoRAs. Additionally, the mixed-task nature of downstream requests necessitates personalized services. To address these challenges, we propose Retrieval-Augmented Mixture of LoRA Experts (RAMoLE), a framework that adaptively retrieves and composes multiple LoRAs based on input prompts. RAMoLE has three main components: LoraRetriever for identifying and retrieving relevant LoRAs, an on-the-fly MoLE mechanism for coordinating the retrieved LoRAs, and efficient batch inference for handling heterogeneous requests. Experimental results show that RAMoLE consistently outperforms baselines, highlighting its effectiveness and scalability.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 42 of its 55 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

42 of 55 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
30 more
  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  4. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  5. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  6. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  7. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  8. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  9. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  10. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  11. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  13. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  14. 2022Task-aware Retrieval with Instructionscited by 107
  15. 2024Higher Layers Need More LoRA Expertscited by 101
  16. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  17. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  18. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  19. 2024Multi-LoRA Composition for Image Generationcited by 80
  20. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  21. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  22. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  23. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  24. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55
  25. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  26. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  27. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  28. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  29. 2024Stylus: Automatic Adapter Selection for Diffusion Modelscited by 18
  30. 2022Efficiently Tuned Parameters are Task Embeddingscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  3. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  4. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answeringcited by 36
  5. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  6. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  7. 2025Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptationcited by 3
  8. 2025Instruction-Guided Autoregressive Neural Network Parameter Generationcited by 3
  9. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  10. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  11. 2026Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memorycited by 2
  12. 2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1
2 more
  1. 2025Tensorized Clustered LoRA Merging for Multi-Task Interferencecited by 1
  2. 2026Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databasescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LoRA Is Slower Than You Think
  2. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
  3. 2025LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
  4. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs
  5. 2025Uni-LoRA: One Vector is All You Need
  6. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
  7. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysis
  8. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning