mlsanityData to 7 Oct 2026

MoVA: Adapting Mixture of Vision Experts to Multimodal Context

arXiv 2404.13046 ↗First posted 19 Apr 2024cs.CVcited by 82

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM). In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 84 of its 112 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

84 of 112 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models12 Mar
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  4. 2023VILA: On Pre-training for Visual Language Models12 Dec
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  6. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2019Towards VQA Models That Can Readcited by 1,696
  13. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  14. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  15. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  16. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  17. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  18. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  19. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  20. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  21. 2016Modeling Context in Referring Expressionscited by 1,054
  22. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  23. 2016A Diagram Is Worth A Dozen Imagescited by 793
  24. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  25. 2024Yi: Open Foundation Models by 01.AIcited by 706
  26. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  27. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  28. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  29. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  30. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  31. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  32. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  33. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  34. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  35. 2021InfographicVQAcited by 430
  36. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  37. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  38. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 42; 2026: 13.

82 papers.

Sort:

By area:

  1. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  2. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  3. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  4. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  5. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  6. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  7. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  8. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  9. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  10. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 72
  11. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  12. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 53
38 more
  1. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 47
  2. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  3. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  4. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  5. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  6. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  7. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  8. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  9. 2024EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLMcited by 25
  10. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  11. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  12. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Drivingcited by 20
  13. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  14. 2025Mixture of Experts in Large Language Modelscited by 15
  15. 2024Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Expertscited by 14
  16. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  17. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Modelscited by 12
  18. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  19. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  20. 2025The Photographer Eye: Teaching Multimodal Large Language Models to Understand Image Aesthetics like Photographerscited by 12
  21. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  22. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  23. 2024Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehensioncited by 10
  24. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  25. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  26. 2025BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perceptioncited by 9
  27. 2024LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformercited by 8
  28. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  29. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
  30. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  31. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Modelscited by 7
  32. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  33. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 6
  34. 2025CoFFT: Chain of Foresight-Focus Thought for Visual Language Modelscited by 6
  35. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  36. 2024Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Modelscited by 5
  37. 2025Enhancing Vision Foundation Models via Multimodal Continual Pre-Trainingcited by 5
  38. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  2. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
  3. 2024MouSi: Poly-Visual-Expert Vision-Language Models
  4. 2025MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention
  5. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
  6. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
  7. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  8. 2023Scaling Vision-Language Models with Sparse Mixture of Experts