mlsanityData to 14 Oct 2026

Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

arXiv 2408.15998 ↗First posted 28 Aug 2024cs.CV cs.AI cs.LGcited by 139

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.

Key facts. Cited by 139 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 90 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

73 of 90 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context19 Apr
  4. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  5. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models5 Mar
  6. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models6 Jan
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  8. 2023VILA: On Pre-training for Visual Language Models12 Dec
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  10. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  11. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
6 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023A Survey on Multimodal Large Language Models23 Jun
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Segment Anythingcited by 8,234
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2022A ConvNet for the 2020scited by 3,579
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,204
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  14. 2016A Diagram Is Worth A Dozen Imagescited by 804
  15. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  16. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  17. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  18. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  19. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  20. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  21. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  22. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  23. 2021OCR-free Document Understanding Transformercited by 407
  24. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  25. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  26. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  27. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  28. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  29. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  30. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  31. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  32. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  33. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  34. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 210
  35. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  36. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  37. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  38. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 73; 2026: 34.

139 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  4. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  5. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  6. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  7. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  8. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  9. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  10. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
  11. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  12. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
38 more
  1. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  2. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  3. 2025Scaling Language-Free Visual Representation Learningcited by 66
  4. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 57
  5. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  6. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  7. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  8. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  10. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  11. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 30
  12. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  13. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 28
  14. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  15. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoderscited by 26
  16. 2025Interpretable and Testable Vision Features via Sparse Autoencoderscited by 25
  17. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problemscited by 24
  18. 2024StreamChat: Chatting with Streaming Videocited by 24
  19. 2025FaceXBench: Evaluating Multimodal LLMs on Face Understandingcited by 24
  20. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 24
  21. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  22. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehensioncited by 23
  23. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 21
  24. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  25. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  26. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  27. 2024Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusioncited by 18
  28. 2024Phantom of Latent for Large Language and Vision Modelscited by 17
  29. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  30. 2025Vision Function Layer in Multimodal LLMscited by 17
  31. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  32. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  33. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  34. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Modelscited by 15
  35. 2025FaceLLM: A Multimodal Large Language Model for Face Understandingcited by 15
  36. 2024Do Language Models Understand Time?cited by 14
  37. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 14
  38. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 13

50 most cited of 139 All 139 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
  2. 2025Rethinking Visual Information Processing in Multimodal LLMs
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models
  6. 2026TGIF: Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs
  7. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  8. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception