mlsanityData to 7 Oct 2026

MouSi: Poly-Visual-Expert Vision-Language Models

arXiv 2401.17221 ↗First posted 30 Jan 2024cs.CV cs.AI cs.CLcited by 24

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current large vision-language models (VLMs) often encounter challenges such as insufficient capabilities of a single visual component and excessively long visual tokens. These issues can limit the model's effectiveness in accurately interpreting complex visual information and over-lengthy contextual information. Addressing these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes the use of ensemble experts technique to synergizes the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc. This technique introduces a fusion network to unify the processing of outputs from different visual experts, while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to alleviate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient and manageable 64 or even down to 1. Experimental results demonstrate that VLMs with multiple experts exhibit consistently superior performance over isolated visual encoders and mark a significant performance boost as more experts are integrated. We have open-sourced the training code used in this report. All of these resources can be found on our project website.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 56 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

45 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
30 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  14. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  15. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  16. 2019Attention is not not Explanationcited by 629
  17. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  18. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  19. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  20. 2018nocaps: novel object captioning at scalecited by 444
  21. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  22. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  23. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  24. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  25. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  26. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  27. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  28. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  29. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  30. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  2. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  3. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  4. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  5. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 21
  6. 2024Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusioncited by 18
  7. 2024Visually Descriptive Language Model for Vector Graphics Reasoningcited by 14
  8. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  9. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  10. 2025Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbationscited by 7
  11. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMscited by 6
  12. 2025LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Expertscited by 4
12 more
  1. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 4
  2. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  3. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuningcited by 3
  4. 2025M3: 3D-Spatial MultiModal Memorycited by 3
  5. 2025Window Token Concatenation for Efficient Visual Large Language Modelscited by 3
  6. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  7. 2026ChartE3: A Comprehensive Benchmark for End-to-End Chart Editingcited by 3
  8. 2025SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generationcited by 2
  9. 2025AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Modelscited by 2
  10. 2025The Role of Entropy in Visual Grounding: Analysis and Optimizationcited by 1
  11. 2025MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attentioncited by 0
  12. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  2. 2025Visual Large Language Models for Generalized and Specialized Applications
  3. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling
  4. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context
  5. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
  6. 2024An Introduction to Vision-Language Modeling
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models