Data to 7 Oct 2026

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

arXiv 2401.16160 ↗First posted 29 Jan 2024cs.CVcited by 84

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more

Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Instruction finetuning on a variety of image-text instruction data is the key to obtaining a versatile Multimodal Large Language Model (MLLM), and different configurations of the instruction data can lead to finetuned models with different capabilities. However, we have discovered that data conflicts are inevitable when mixing instruction data from distinct domains, which can result in performance drops for tasks of a specific domain. To address this issue, we propose to apply an efficient Mixture of Experts (MoE) design, which is a sparse Mixture of LoRA Experts (MoLE) for instruction finetuning MLLMs. Within the Transformer layers, we extend the popular Low-Rank Adaption (LoRA) method by creating a set of LoRA experts specifically for the MLP layer, and route each token to the top-1 expert based on a routing function, allowing adaptive choices for tokens from different domains. Since the LoRA experts are sparsely activated, the training and inference cost are kept roughly constant compared to the original LoRA method. By replacing the plain-LoRA of LLaVA-1.5 with our MoE design, our final model is named LLaVA-MoLE. Extensive experiments proved that LLaVA-MoLE effectively mitigates the data conflict issue when mixing multiple distinct instruction datasets with various configurations, and achieves consistent performance gains over the strong plain-LoRA baselines. Most importantly, on the mixed datasets, LLaVA-MoLE can even outperform the plain-LoRA baseline trained with twice the samples.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 51 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

42 of 51 references.

Closest ancestors · 7

Same direction, newest first

Other references · 35

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Language Models are Few-Shot Learnerscited by 32,199
2023GPT-4 Technical Reportcited by 18,132
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2023Visual Instruction Tuningcited by 7,792
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
2024Mixtral of Expertscited by 1,785
2019Towards VQA Models That Can Readcited by 1,696

23 more

2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
2021Scaling Vision with Sparse Mixture of Expertscited by 740
2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
2021InfographicVQAcited by 430
2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
2023SVIT: Scaling up Visual Instruction Tuningcited by 161
2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
2021Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layoutscited by 82
2020Pathological Visual Question Answeringcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 36; 2026: 14.

84 papers.

Sort:

By area:

2024A Survey on Mixture of Experts in Large Language Modelscited by 233
2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
2024A Survey on LoRA of Large Language Modelscited by 67
2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27

38 more

2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24
2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEcited by 18
2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
2025MoE-Loco: Mixture of Experts for Multitask Locomotioncited by 17
2024PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRAcited by 16
2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Datacited by 14
2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 14
2024WalkVLM:Aid Visually Impaired People Walking by Vision Language Modelcited by 13
2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Modelscited by 12
2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
2024RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Modelscited by 11
2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
2025Ensembles of Low-Rank Expert Adapterscited by 9
2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8
2025CoPL: Collaborative Preference Learning for Personalizing LLMscited by 8
2025Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detectioncited by 8
2025Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extractioncited by 7
2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modulescited by 7
2025Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safetycited by 7
2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
2025MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidancecited by 6
2025MokA: Multimodal Low-Rank Adaptation for MLLMscited by 6
2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
2024FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Modelscited by 5
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
2024Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistantscited by 5
2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5
2025Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detectioncited by 5

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way