mlsanityData to 7 Oct 2026

MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

arXiv 2405.13053 ↗First posted 19 May 2024cs.CL cs.AIcited by 14

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

The pretrain+fine-tune paradigm is foundational for deploying large language models (LLMs) across various downstream applications. Within this framework, Low-Rank Adaptation (LoRA) stands out for its parameter-efficient fine-tuning (PEFT), producing numerous reusable task-specific LoRA adapters. However, this approach requires explicit task intention selection, posing challenges for autonomous task sensing and switching during inference with multiple existing LoRA adapters embedded in a single LLM. In this work, we introduce MeteoRA (Multiple-tasks embedded LoRA), a scalable and efficient framework that reuses multiple task-specific LoRA adapters into the base LLM via a full-mode Mixture-of-Experts (MoE) architecture. This framework also includes novel MoE forward acceleration strategies to address the efficiency challenges of traditional MoE implementations. Our evaluation, using the LlaMA2-13B and LlaMA3-8B base models equipped with 28 existing LoRA adapters through MeteoRA, demonstrates equivalent performance with the traditional PEFT method. Moreover, the LLM equipped with MeteoRA achieves superior performance in handling composite tasks, effectively solving ten sequential problems in a single inference pass, thereby demonstrating the framework's enhanced capability for timely adapter switching.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 32 of its 47 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

32 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  2. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  10. 2023BloombergGPT: A Large Language Model for Financecited by 923
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
18 more
  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  3. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  4. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  5. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  6. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  7. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  8. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  9. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  10. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  11. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  12. 2020Improving model calibration with accuracy versus uncertainty optimizationcited by 123
  13. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  14. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  15. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  16. 2022Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learningcited by 11
  17. 2019Operational Calibration: Debugging Confidence Errors for DNNs in the Fieldcited by 6
  18. 2023Anatomy of High-Performance GEMM with Online Fault Tolerance on GPUscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  3. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  4. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  5. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  6. 2024DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Modelscited by 2
  7. 2025Memorization and Knowledge Injection in Gated LLMscited by 2
  8. 2026The Appeal and Reality of Recycling LoRAs with Adaptive Mergingcited by 2
  9. 2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1
  10. 2026RobuMTL: Enhancing Multi-Task Learning Robustness Against Weather Conditionscited by 1
  11. 2026Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Modelcited by 0
  12. 2026Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answeringcited by 0
2 more
  1. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  2. 2026ElectrolyteFM: Unifying Electrolyte Property Prediction through Cross-Property Knowledge Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
  2. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  3. 2023Token-Level Adaptation of LoRA Adapters for Downstream Task Generalization
  4. 2025LoRA Is Slower Than You Think
  5. 2024MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
  6. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
  7. 2025MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
  8. 2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning