mlsanityData to 7 Oct 2026

Multimodal Instruction Tuning with Conditional Mixture of LoRA

arXiv 2402.15896 ↗First posted 24 Feb 2024cs.CVcited by 36

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in diverse tasks across different domains, with an increasing focus on improving their zero-shot generalization capabilities for unseen multimodal tasks. Multimodal instruction tuning has emerged as a successful strategy for achieving zero-shot generalization by fine-tuning pre-trained models on diverse multimodal tasks through instructions. As MLLMs grow in complexity and size, the need for parameter-efficient fine-tuning methods like Low-Rank Adaption (LoRA), which fine-tunes with a minimal set of parameters, becomes essential. However, applying LoRA in multimodal instruction tuning presents the challenge of task interference, which leads to performance degradation, especially when dealing with a broad array of multimodal tasks. To address this, this paper introduces a novel approach that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance, aiming to mitigate task interference. Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks, demonstrating its efficacy and adaptability in diverse multimodal tasks.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 48 references that are in the map. First posted on arXiv 24 Feb 2024.

Builds on

34 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
20 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  4. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  5. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  6. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  7. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  8. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  9. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  11. 2019Scene Text Visual Question Answeringcited by 327
  12. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  13. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  14. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  15. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  16. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  17. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  18. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  19. 2019Many Task Learning with Task Routingcited by 50
  20. 2019Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernelscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 15; 2026: 10.

36 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
  3. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 29
  4. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  5. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  6. 2024Plug-and-Play Grounding of Reasoning in Multimodal Large Language Modelscited by 18
  7. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learningcited by 16
  8. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  9. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  10. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  11. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  12. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
24 more
  1. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  2. 2026VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuningcited by 6
  3. 2024Advancing Chart Question Answering with Robust Chart Component Recognitioncited by 5
  4. 2025Pisces: An Auto-regressive Foundation Model for Image Understanding and Generationcited by 5
  5. 2025Large Multimodal Models for Low-Resource Languages: A Surveycited by 4
  6. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  7. 2025ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptationcited by 3
  8. 2025GLAD: Generalizable Tuning for Vision-Language Modelscited by 3
  9. 2024AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discoverycited by 2
  10. 2024Error-driven Data-efficient Large Multimodal Model Tuningcited by 2
  11. 2025Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist Systemscited by 2
  12. 2025Mixture-of-Experts for Personalized and Semantic-Aware Next Location Predictioncited by 2
  13. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  14. 2025MoRA: Missing Modality Low-Rank Adaptation for Visual Recognitioncited by 2
  15. 2026MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translationcited by 2
  16. 2025NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptationcited by 1
  17. 2026AuRA: Internalizing Audio Understanding into LLMs as LoRAcited by 1
  18. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtentioncited by 1
  19. 2026DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learningcited by 0
  20. 2026Multimodal Task Interference: A Benchmark and Analysis of History-Target Mismatch in Multimodal LLMscited by 0
  21. 2026Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atomscited by 0
  22. 2026Dense Coordinate-List Fine-Tuning Induces a Controllable Interference Surface in Vision-Language Modelscited by 0
  23. 2026LoRA-Based Cascaded Multimodal Fusion for Action Recognition in Medical Training Environmentscited by 0
  24. 2026LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs
  2. 2024CoMMIT: Coordinated Multimodal Instruction Tuning
  3. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
  4. 2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning
  5. 2025LoRA Is Slower Than You Think
  6. 2025AsymLoRA: Harmonizing Data Conflicts and Commonalities in MLLMs
  7. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models
  8. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?