mlsanityData to 7 Oct 2026

LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild

arXiv 2402.09997 ↗First posted 15 Feb 2024cs.AI cs.CL cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) provides an effective yet efficient solution for fine-tuning large language models (LLM). The modular and plug-and-play nature of LoRA enables the integration of diverse domain-specific LoRAs to enhance the capabilities of LLMs. Previous research on exploiting multiple LoRAs either focuses on specific isolated downstream tasks or fixes the selection of LoRAs during training. However, in real-world scenarios, LLMs receive diverse prompts covering different tasks, and the pool of candidate LoRAs is often dynamically updated. To bridge this gap, we propose LoraRetriever, a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts. LoraRetriever contains three main components: firstly, identifying and retrieving LoRAs relevant to the given input; secondly, formulating strategies for effectively integrating the retrieved LoRAs; and thirdly, developing efficient batch inference to accommodate heterogeneous requests. Experimental results indicate that LoraRetriever consistently outperforms the baselines, highlighting its practical effectiveness and versatility.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 23 of its 45 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

23 of 45 references.

References · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  10. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  11. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  12. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
11 more
  1. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  2. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  3. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  4. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  5. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  6. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  7. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  8. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  9. 2023Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaborationcited by 52
  10. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  11. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 30; 2026: 16.

68 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  3. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  4. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  5. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  6. 2024A Survey on LoRA of Large Language Modelscited by 67
  7. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  8. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  9. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  10. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  11. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  12. 2024LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystemcited by 22
38 more
  1. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  2. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  3. 2024Stylus: Automatic Adapter Selection for Diffusion Modelscited by 18
  4. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  5. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  6. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  7. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  8. 2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
  9. 2025Nature-Inspired Population-Based Evolution of Large Language Modelscited by 14
  10. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  11. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  12. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  13. 2024T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Conceptscited by 12
  14. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  15. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  16. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
  17. 2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
  18. 2025Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMscited by 9
  19. 2025MEraser: An Effective Fingerprint Erasure Approach for Large Language Modelscited by 9
  20. 2024An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routingcited by 7
  21. 2025Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentationcited by 7
  22. 2026Effective LoRA Adapter Routing using Task Representationscited by 6
  23. 2025Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Trainingcited by 5
  24. 2025Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generationcited by 5
  25. 2025Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptationcited by 5
  26. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 4
  27. 2025HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalizationcited by 4
  28. 2025Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasingcited by 3
  29. 2025LoRA on the Go: Instance-level Dynamic LoRA Selection and Mergingcited by 3
  30. 2026Compress then Merge: From Multiple LoRAs into One Low-Rank Adaptercited by 3
  31. 2024Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projectioncited by 2
  32. 2025Memorization and Knowledge Injection in Gated LLMscited by 2
  33. 2025qa-FLoRA: Data-free query-adaptive Fusion of LoRAs for LLMscited by 2
  34. 2026Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memorycited by 2
  35. 2024SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learningcited by 1
  36. 2024CopRA: A Progressive LoRA Training Strategycited by 1
  37. 2025MedForge: Building Medical Foundation Models Like Open Source Software Developmentcited by 1
  38. 2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1

50 most cited of 68 All 68 (JSON)