# RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts

arXiv: https://arxiv.org/abs/2412.05679 · HTML: https://mlsanity.com/paper/2412.05679/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 97 references that are in the map.
- First posted on arXiv 7 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 39)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2110.08733/) (2021)
- [RSGPT: A Remote Sensing Vision Language Model and Benchmark](https://mlsanity.com/paper/2307.15266/) (2023)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2309.14717/) (2023)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)
- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2406.12384/) (2024)
- [SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model](https://mlsanity.com/paper/2401.09712/) (2024)
- [ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference](https://mlsanity.com/paper/2407.12442/) (2024)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding](https://mlsanity.com/paper/2406.10100/) (2024)
- [VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis](https://mlsanity.com/paper/2403.20213/) (2024)
- [NExT-Chat: An LMM for Chat, Detection and Segmentation](https://mlsanity.com/paper/2311.04498/) (2023)

## Built on by (most cited first, 29 of 29)

- [SegEarth-R1: Geospatial Pixel Reasoning via Large Language Model](https://mlsanity.com/paper/2504.09644/) (2025)
- [When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning](https://mlsanity.com/paper/2503.07588/) (2025)
- [EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLM](https://mlsanity.com/paper/2506.01667/) (2025)
- [REOBench: Benchmarking Robustness of Earth Observation Foundation Models](https://mlsanity.com/paper/2505.16793/) (2025)
- [GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes](https://mlsanity.com/paper/2511.22645/) (2025)
- [Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality](https://mlsanity.com/paper/2603.00988/) (2026)
- [SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2512.20013/) (2025)
- [GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2602.14201/) (2026)
- [EagleVision: Object-level Attribute Multimodal LLM for Remote Sensing](https://mlsanity.com/paper/2503.23330/) (2025)
- [RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs](https://mlsanity.com/paper/2604.07765/) (2026)
- [SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts](https://mlsanity.com/paper/2512.02517/) (2025)
- [Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensing](https://mlsanity.com/paper/2512.19302/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Images](https://mlsanity.com/paper/2508.05202/) (2025)
- [CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts](https://mlsanity.com/paper/2509.14104/) (2025)
- [Co-Training Vision Language Models for Remote Sensing Multi-task Learning](https://mlsanity.com/paper/2511.21272/) (2025)
- [FUSE-RSVLM: Feature Fusion Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2512.24022/) (2025)
- [Remote Sensing Large Vision-Language Model: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling](https://mlsanity.com/paper/2506.21863/) (2025)
- [RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observation](https://mlsanity.com/paper/2604.17243/) (2026)
- [RemoteZero: Geospatial Reasoning with Zero Labels](https://mlsanity.com/paper/2605.04451/) (2026)
- [WeaveEarth: Structured Evidence Construction and Reasoning for Training-Free UHR Remote Sensing Understanding](https://mlsanity.com/paper/2607.10120/) (2026)
- [Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges](https://mlsanity.com/paper/2508.06832/) (2025)
- [Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding](https://mlsanity.com/paper/2604.11122/) (2026)
- [Decoding the Delta: Unifying Remote Sensing Change Detection and Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2604.14044/) (2026)
- [Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysis](https://mlsanity.com/paper/2601.14637/) (2026)
- [Mixture-of-Experts in Remote Sensing: A Survey](https://mlsanity.com/paper/2604.03342/) (2026)
- [SenseBench: A Benchmark for Remote Sensing Low-Level Visual Perception and Description in Large Vision-Language Models](https://mlsanity.com/paper/2605.10576/) (2026)
- [SkyVLaM: Multimodal Large Language Model for UAV Video Understanding in Remote Sensing](https://mlsanity.com/paper/2607.17386/) (2026)
- [SA-GEM: Scale-Adaptive and Geospatial Evidence-Modulated Token Pruning for Efficient Remote Sensing Large Vision-Language Models](https://mlsanity.com/paper/2608.15075/) (2026)

## Abstract (arXiv)

Remote Sensing Vision-Language Models (RS VLMs) have made much progress in the tasks of remote sensing (RS) image comprehension. While performing well in multi-modal reasoning and multi-turn conversations, the existing models lack pixel-level understanding and struggle with multi-image inputs. In this work, we propose RSUniVLM, a unified, end-to-end RS VLM designed for comprehensive vision understanding across multiple granularity, including image-level, region-level, and pixel-level tasks. RSUniVLM also performs effectively in multi-image analysis, with instances of change detection and change captioning. To enhance the model's ability to capture visual information at different levels without increasing model size, we design a novel architecture called Granularity-oriented Mixture of Experts to constraint the model to about 1 billion parameters. We also construct a large-scale RS instruction-following dataset based on a variety of existing datasets in both RS and general domain, encompassing various tasks such as object localization, visual question answering, and semantic segmentation. Substantial experiments have been conducted to validate the superiority of the proposed RSUniVLM up to state-of-the-art across various RS tasks. Code and model will be available at hrefhttps://github.com/xuliu-cyber/RSUniVLMhere.