mlsanityData to 7 Oct 2026

MoAI: Mixture of All Intelligence for Large Language and Vision Models

arXiv 2403.07508 ↗First posted 12 Mar 2024cs.CVcited by 36

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rise of large language models (LLMs) and instruction tuning has led to the current trend of instruction-tuned large language and vision models (LLVMs). This trend involves either meticulously curating numerous instruction tuning datasets tailored to specific objectives or enlarging LLVMs to manage vast amounts of vision language (VL) data. However, current LLVMs have disregarded the detailed and comprehensive real-world scene understanding available from specialized computer vision (CV) models in visual perception tasks such as segmentation, detection, scene graph generation (SGG), and optical character recognition (OCR). Instead, the existing LLVMs rely mainly on the large capacity and emergent capabilities of their LLM backbones. Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI operates through two newly introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the outputs of the external CV models, the MoAI-Compressor aligns and condenses them to efficiently use relevant auxiliary visual information for VL tasks. MoAI-Mixer then blends three types of intelligence (1) visual features, (2) auxiliary features from the external CV models, and (3) language features by utilizing the concept of Mixture of Experts. Through this integration, MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot VL tasks, particularly those related to real-world scene understanding such as object existence, positions, relations, and OCR without enlarging the model size or curating extra visual instruction tuning datasets.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 72 of its 100 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

72 of 100 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2023Segment Anythingcited by 8,158
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  13. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  14. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  15. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  16. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  17. 2020Learning to summarize from human feedbackcited by 2,562
  18. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  19. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  20. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  21. 2024Mixtral of Expertscited by 1,785
  22. 2019Towards VQA Models That Can Readcited by 1,696
  23. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  24. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  25. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  26. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  27. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  28. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  29. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  30. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  31. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  32. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  33. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  34. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  35. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  36. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  37. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  38. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 13; 2026: 9.

36 papers.

Sort:

By area:

  1. 2024What matters when building vision-language models?cited by 321
  2. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  3. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  4. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  5. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  6. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  7. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  8. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  9. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  10. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  11. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  12. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
24 more
  1. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  2. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  3. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  4. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  5. 2025DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenescited by 10
  6. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  7. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  8. 2024SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Modelscited by 8
  9. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  10. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  11. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5
  12. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  13. 2026Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Taskscited by 4
  14. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  15. 2026Recursive Think-Answer Process for LLMs and VLMscited by 2
  16. 2024Intriguing Properties of Large Language and Vision Modelscited by 1
  17. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  18. 2025Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixturecited by 0
  19. 2026Robust Grounding with MLLMs Against Occlusion and Small Objects via Language-Guided Semantic Cuescited by 0
  20. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  21. 2026Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMscited by 0
  22. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  23. 2026Mixture of Cognitive Experts in Large Vision-Language Modelscited by 0
  24. 2026Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention
  2. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  4. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
  5. 2023Scaling Vision-Language Models with Sparse Mixture of Experts
  6. 2024A Survey on Mixture of Experts in Large Language Models
  7. 2024MouSi: Poly-Visual-Expert Vision-Language Models
  8. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks