mlsanityData to 7 Oct 2026

InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding

arXiv 2403.01487 ↗First posted 3 Mar 2024cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have experienced significant advancements recently. Nevertheless, challenges persist in the accurate recognition and comprehension of intricate details within high-resolution images. Despite being indispensable for the development of robust MLLMs, this area remains underinvestigated. To tackle this challenge, our work introduces InfiMM-HD, a novel architecture specifically designed for processing images of different resolutions with low computational overhead. This innovation facilitates the enlargement of MLLMs to higher-resolution capabilities. InfiMM-HD incorporates a cross-attention module and visual windows to reduce computation costs. By integrating this architectural design with a four-stage training pipeline, our model attains improved visual perception efficiently and cost-effectively. Empirical study underscores the robustness and effectiveness of InfiMM-HD, opening new avenues for exploration in related areas. Codes and models can be found at https://huggingface.co/Infi-MM/infimm-hd

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 39 references that are in the map. First posted on arXiv 3 Mar 2024.

Builds on

23 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning10 Jan
  2. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models20 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
7 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  3. 2019Scene Text Visual Question Answeringcited by 327
  4. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  5. 2020Captioning Images Taken by People Who Are Blindcited by 108
  6. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  7. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Modelscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 1.

11 papers.

  1. 2024What matters when building vision-language models?cited by 321
  2. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  3. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  4. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  5. 2024EVLM: An Efficient Vision-Language Model for Visual Understandingcited by 23
  6. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  7. 2024Law of Vision Representation in MLLMscited by 16
  8. 2025Slow-Fast Architecture for Video Multi-Modal Large Language Modelscited by 10
  9. 2025Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraintscited by 6
  10. 2025A Training-Free, Task-Agnostic Framework for Enhancing MLLM Performance on High-Resolution Imagescited by 4
  11. 2026Towards High-Resolution Visual Perception via Hierarchical Entity Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2024VisualRWKV-HD and UHD: Advancing High-Resolution Processing for Visual Language Models
  5. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  6. 2024AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understanding
  7. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites