mlsanityData to 7 Oct 2026

Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning

arXiv 2312.12379 ↗First posted 19 Dec 2023cs.CVcited by 106

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Instruction tuning of Large Vision-language Models (LVLMs) has revolutionized the development of versatile models with zero-shot generalization across a wide range of downstream vision-language tasks. However, the diversity of training tasks of different sources and formats would lead to inevitable task conflicts, where different tasks conflict for the same set of model parameters, resulting in sub-optimal instruction-following abilities. To address that, we propose the Mixture of Cluster-conditional LoRA Experts (MoCLE), a novel Mixture of Experts (MoE) architecture designed to activate the task-customized model parameters based on the instruction clusters. A separate universal expert is further incorporated to improve generalization capabilities of MoCLE for novel instructions. Extensive experiments on InstructBLIP and LLaVA demonstrate the effectiveness of MoCLE.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 113 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

75 of 113 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  8. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar

Other references · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  14. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  15. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  16. 2023LIMA: Less Is More for Alignmentcited by 1,080
  17. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  18. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  19. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  20. 2017A Survey on Multi-Task Learningcited by 838
  21. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  22. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  23. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  24. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  25. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  26. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  27. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  28. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  31. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  32. 2016Visual Dialogcited by 377
  33. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  34. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  35. 2022Visual Spatial Reasoningcited by 288
  36. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  37. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  38. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 52; 2025: 31; 2026: 23.

106 papers.

Sort:

By area:

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  3. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  4. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  5. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  6. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  7. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  8. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  9. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  10. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  11. 2024A Survey on LoRA of Large Language Modelscited by 67
  12. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
38 more
  1. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  2. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  3. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
  4. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  5. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  6. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  7. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 33
  8. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  9. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  10. 2024A Survey of Small Language Modelscited by 31
  11. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  12. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  13. 2024Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparatorscited by 19
  14. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  15. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  16. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  17. 2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-designcited by 18
  18. 2024EEGMamba: Bidirectional State Space Model with Mixture of Experts for EEG Multi-task Classificationcited by 18
  19. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEcited by 18
  20. 2025MoE-Loco: Mixture of Experts for Multitask Locomotioncited by 17
  21. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
  22. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  23. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  24. 2024Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Mergingcited by 14
  25. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 13
  26. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  27. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  28. 2024RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Modelscited by 11
  29. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  30. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
  31. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  32. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  33. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  34. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  35. 2025Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learningcited by 8
  36. 2025Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detectioncited by 8
  37. 2024XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Expertscited by 7
  38. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  2. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Models
  3. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
  4. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models
  5. 2024Less is More: High-value Data Selection for Visual Instruction Tuning
  6. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
  7. 2024Rethinking Overlooked Aspects in Vision-Language Models
  8. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality