mlsanityData to 7 Oct 2026

Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

arXiv 2403.03003 ↗First posted 5 Mar 2024cs.CVcited by 112

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Despite remarkable progress, existing multimodal large language models (MLLMs) are still inferior in granular visual recognition. Contrary to previous works, we study this problem from the perspective of image resolution, and reveal that a combination of low- and high-resolution visual features can effectively mitigate this shortcoming. Based on this observation, we propose a novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation (MRA). In particular, MRA adopts two visual pathways for images with different resolutions, where high-resolution visual information is embedded into the low-resolution pathway via the novel mixture-of-resolution adapters (MR-Adapters). This design also greatly reduces the input sequence length of MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the new model LLaVA-HR. We conduct extensive experiments on 11 vision-language (VL) tasks, which show that LLaVA-HR outperforms existing MLLMs on 8 VL tasks, e.g., +9.4% on TextVQA. More importantly, both training and inference of LLaVA-HR remain efficient with MRA, e.g., 20 training hours and 3× inference speed than LLaVA-1.5. Source codes are released at: https://github.com/luogen1996/LLaVA-HR.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 41 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

31 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
12 more
  1. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  7. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  8. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  9. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  10. 2019Multimodal Transformer with Multi-View Visual Representation for Image Captioningcited by 86
  11. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillingscited by 68
  12. 2022A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehensioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 51; 2026: 18.

112 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  3. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  4. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  5. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  6. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  7. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  8. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  9. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  10. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 92
  11. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  12. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
38 more
  1. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  2. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  3. 2025Kwai Keye-VL Technical Reportcited by 47
  4. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  5. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  6. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  7. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  8. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  9. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  10. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  11. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  12. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  13. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  14. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
  15. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  16. 2024CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenariocited by 27
  17. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  18. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  19. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 26
  20. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  21. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  22. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  23. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  24. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  25. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 21
  26. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 21
  27. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimmingcited by 20
  28. 2025FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registerscited by 20
  29. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuningcited by 19
  30. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinementcited by 18
  31. 2024HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Modelscited by 16
  32. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  33. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  34. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  35. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 13
  36. 2024γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Modelscited by 13
  37. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Modelscited by 13
  38. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models
  2. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
  3. 2024AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understanding
  4. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  6. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
  7. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
  8. 2023InfMLLM: A Unified Framework for Visual-Language Tasks