mlsanityData to 14 Oct 2026

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

arXiv 2312.00968 ↗First posted 1 Dec 2023cs.CV cs.CLcited by 26

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks. However, generalist LMMs often suffer from performance degradation when tuned over a large collection of tasks. Recent research suggests that Mixture of Experts (MoE) architectures are useful for instruction tuning, but for LMMs of parameter size around O(50-100B), the prohibitive cost of replicating and storing the expert models severely limits the number of experts we can use. We propose Omni-SMoLA, an architecture that uses the Soft MoE approach to (softly) mix many multimodal low rank experts, and avoids introducing a significant number of new parameters compared to conventional MoE models. The core intuition here is that the large model provides a foundational backbone, while different lightweight experts residually learn specialized knowledge, either per-modality or multimodally. Extensive experiments demonstrate that the SMoLA approach helps improve the generalist performance across a broad range of generative vision-and-language tasks, achieving new SoTA generalist performance that often matches or outperforms single specialized LMM baselines, as well as new SoTA specialist performance.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 58 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

49 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling18 Oct
  2. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial26 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
34 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  12. 2016A Diagram Is Worth A Dozen Imagescited by 804
  13. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  15. 2017Learning multiple visual domains with residual adapterscited by 557
  16. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  17. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  18. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  19. 2021InfographicVQAcited by 437
  20. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  22. 2019Scene Text Visual Question Answeringcited by 331
  23. 2023From Sparse to Soft Mixtures of Expertscited by 248
  24. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  25. 2019Pareto Multi-Task Learningcited by 224
  26. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  27. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  28. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  29. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  30. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  31. 2020Captioning Images Taken by People Who Are Blindcited by 110
  32. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  33. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  34. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 9; 2026: 7.

26 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  3. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  4. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  5. 2024Foundation Models for Video Understanding: A Surveycited by 46
  6. 2024Chart-based Reasoning: Transferring Capabilities from LLMs to VLMscited by 20
  7. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  8. 2025Mixture of Experts in Large Language Modelscited by 15
  9. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understandingcited by 12
  10. 2024VA-MoE: Variables-Adaptive Mixture of Experts for Incremental Weather Forecastingcited by 6
  11. 2024Revisiting Multi-Modal LLM Evaluationcited by 5
  12. 2025CAMEx: Curvature-aware Merging of Expertscited by 5
14 more
  1. 2025Where is this coming from? Making groundedness count in the evaluation of Document VQA modelscited by 4
  2. 2025LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Expertscited by 4
  3. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  4. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 4
  5. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Expertscited by 4
  6. 2026Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognitioncited by 3
  7. 2025Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classificationcited by 2
  8. 2025VIKSER: Visual Knowledge-Driven Self-Reinforcing Reasoning Frameworkcited by 1
  9. 2026LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspectivecited by 1
  10. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  11. 2026Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encodercited by 0
  12. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  13. 2026CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Predictioncited by 0
  14. 2026Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
  2. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
  3. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  4. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  5. 2025SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
  6. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
  7. 2026Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
  8. 2023Scaling Vision-Language Models with Sparse Mixture of Experts