mlsanityData to 7 Oct 2026

HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs

arXiv 2506.17608 ↗First posted 21 Jun 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The integration of high-resolution image features in modern multimodal large language models has demonstrated significant improvements in fine-grained visual understanding tasks, achieving high performance across multiple benchmarks. Since these features are obtained from large image encoders like ViT, they come with a significant increase in computational costs due to multiple calls to these encoders. In this work, we first develop an intuition for feature upsampling as a natural extension of high-resolution feature generation. Through extensive experiments and ablations, we demonstrate how a shallow feature enricher can achieve competitive results with tremendous reductions in training and inference time as well as computational cost, with upto 1.5x saving in FLOPs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 11 references that are in the map. First posted on arXiv 21 Jun 2025.

Builds on

9 of 11 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 8

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024FeatUp: A Model-Agnostic Framework for Features at Any Resolutioncited by 114
  7. 2024When Do We Not Need Larger Vision Models?cited by 65
  8. 2024LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT Descriptorscited by 27

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Evaluating Variance in Visual Question Answering Benchmarkscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models
  2. 2025A Training-Free, Task-Agnostic Framework for Enhancing MLLM Performance on High-Resolution Images
  3. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
  4. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding
  5. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models
  6. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
  7. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models