# MGIMM: Multi-Granularity Instruction Multimodal Model for Attribute-Guided Remote Sensing Image Detailed Description

arXiv: https://arxiv.org/abs/2406.04716 · HTML: https://mlsanity.com/paper/2406.04716/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 37 of its 67 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 37)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)
- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://mlsanity.com/paper/2312.00784/) (2023)
- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [On the Exploitability of Instruction Tuning](https://mlsanity.com/paper/2306.17194/) (2023)
- [DetGPT: Detect What You Need via Reasoning](https://mlsanity.com/paper/2305.14167/) (2023)
- [Anchor-free Oriented Proposal Generator for Object Detection](https://mlsanity.com/paper/2110.01931/) (2021)
- [Vision-Language Models in Remote Sensing: Current Progress and Future Trends](https://mlsanity.com/paper/2305.05726/) (2023)
- [Parallel Context Windows for Large Language Models](https://mlsanity.com/paper/2212.10947/) (2022)
- [Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis](https://mlsanity.com/paper/2403.19646/) (2024)

## Built on by (most cited first, 3 of 3)

- [Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)](https://mlsanity.com/paper/2503.11070/) (2025)
- [Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives](https://mlsanity.com/paper/2505.14361/) (2025)
- [GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning](https://mlsanity.com/paper/2603.09566/) (2026)

## Abstract (arXiv)

Recently, large multimodal models have built a bridge from visual to textual information, but they tend to underperform in remote sensing scenarios. This underperformance is due to the complex distribution of objects and the significant scale differences among targets in remote sensing images, leading to visual ambiguities and insufficient descriptions by these multimodal models. Moreover, the lack of multimodal fine-tuning data specific to the remote sensing field makes it challenging for the model's behavior to align with user queries. To address these issues, this paper proposes an attribute-guided Multi-Granularity Instruction Multimodal Model (MGIMM) for remote sensing image detailed description. MGIMM guides the multimodal model to learn the consistency between visual regions and corresponding text attributes (such as object names, colors, and shapes) through region-level instruction tuning. Then, with the multimodal model aligned on region-attribute, guided by multi-grain visual features, MGIMM fully perceives both region-level and global image information, utilizing large language models for comprehensive descriptions of remote sensing images. Due to the lack of a standard benchmark for generating detailed descriptions of remote sensing images, we construct a dataset featuring 38,320 region-attribute pairs and 23,463 image-detailed description pairs. Compared with various advanced methods on this dataset, the results demonstrate the effectiveness of MGIMM's region-attribute guided learning approach. Code can be available at https://github.com/yangcong356/MGIMM.git