mlsanityData to 14 Oct 2026

LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning

arXiv 2604.02338 ↗First posted 1 Feb 2026cs.LG cs.CL cs.CVcited by 0

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

MoE-PEFT methods combine Mixture of Experts with parameter-efficient fine-tuning for multi-task adaptation, but require separate adapters per expert causing trainable parameters to scale linearly with expert count and limiting applicability to adapter-based architectures. We propose LiME (Lightweight Mixture of Experts), which achieves expert specialization through lightweight modulation rather than adapter replication. Instead of separate adapters, LiME uses a single shared PEFT module and modulates its output with lightweight expert vectors, reducing expert parameters while generalizing to any PEFT method. Notably, LiME introduces zero-parameter routing by leveraging existing frozen and adapted representations eliminating learned router parameters typically required per layer. Theoretically, we prove that (i) more experts preserve more task-relevant information and (ii) modulation approximates full expert-specific PEFT with bounded error. LiME further incorporates n-gram windowed routing and adaptive expert selection (Auto Top-K) based on routing confidence. Experiments on MMT-47, a multimodal multi-task benchmark with 47 tasks spanning text, image, and video, demonstrate that LiME achieves competitive or superior performance while using up to 4x fewer trainable parameters and up to 29% faster training compared to corresponding MoE-PEFT baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 57 of its 80 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

57 of 80 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning9 Jan
  2. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts30 Sep
  3. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism1 Apr
  4. 2024MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning12 Dec
  5. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  6. 2024Higher Layers Need More LoRA Experts13 Feb
  7. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2024Mixtral of Expertscited by 1,823
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  12. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  13. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  14. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  15. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  16. 2024Mixture of LoRA Expertscited by 181
  17. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  18. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  19. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
  20. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  21. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  22. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  23. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  24. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  25. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  26. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  27. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Modelscited by 16
  28. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  29. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  30. 2024MVTamperBench: Evaluating Robustness of Vision-Language Modelscited by 11
  31. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  32. 2025MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learningcited by 9
  33. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
  34. 2023Separate and Locate: Rethink the Text in Text-based Visual Question Answeringcited by 7
  35. 2025Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMscited by 5
  36. 2024Does Self-Attention Need Separate Weights in Transformers?cited by 4
  37. 2025Predicting Through Generation: Why Generation Is Better for Predictioncited by 2
  38. 2025SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networkscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning
  2. 2026Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation
  3. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
  4. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  5. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners
  6. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
  7. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  8. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning