mlsanityData to 7 Oct 2026

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

arXiv 2311.02684 ↗First posted 5 Nov 2023cs.CV cs.CLcited by 25

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Recent studies have demonstrated Large Language Models (LLMs) can extend their zero-shot generalization capabilities to multimodal learning through instruction tuning. As more modalities and downstream tasks are introduced, negative conflicts and interference may have a worse impact on performance. While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called Octavius, for comprehensive studies and experimentation on multimodal learning with Multimodal Large Language Models (MLLMs). Specifically, we combine the well-known Mixture-of-Experts (MoE) and one of the representative PEFT techniques, i.e., LoRA, designing a novel LLM-based decoder, called LoRA-MoE, for multimodal learning. To the best of our knowledge, we are one of the pioneering efforts to introduce MoE into MLLMs to address this problem. The experimental results (about 20% improvement) have shown the effectiveness and versatility of our design in various 2D and 3D downstream tasks. Code and datasets are available at https://openlamm.github.io/tutorial/.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 55 of its 71 references that are in the map. First posted on arXiv 5 Nov 2023.

Builds on

55 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun

Other references · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  13. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  14. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  15. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  16. 2020Point Transformercited by 1,462
  17. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  18. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  19. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  20. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  21. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  22. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  23. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  24. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  25. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  26. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  27. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  28. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  29. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  30. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  31. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  32. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  33. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  34. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  35. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  36. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  37. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  38. 2021FCAF3D: Fully Convolutional Anchor-Free 3D Object Detectioncited by 97

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 13; 2026: 7.

25 papers.

Sort:

By area:

  1. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  2. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
  3. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  4. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  5. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  6. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  7. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  8. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  9. 2025CoPL: Collaborative Preference Learning for Personalizing LLMscited by 8
  10. 2025MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understandingcited by 6
  11. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  12. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
13 more
  1. 2025Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptationcited by 5
  2. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  3. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  4. 2026MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Freecited by 4
  5. 2026CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generationcited by 3
  6. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Modelscited by 2
  7. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
  8. 2026Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMscited by 1
  9. 2026Sparse Spectral LoRA: Routed Experts for Medical VLMscited by 1
  10. 2025Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learningcited by 0
  11. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  12. 2025InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generationcited by 0
  13. 2026GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRA
  2. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs
  3. 2024CoMMIT: Coordinated Multimodal Instruction Tuning
  4. 2025HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
  5. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  6. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  7. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  8. 2024SoupLM: Model Integration in Large Language and Multi-Modal Models