mlsanityData to 7 Oct 2026

SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model

arXiv 2401.09712 ↗First posted 18 Jan 2024cs.CVcited by 112

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large language models (LLMs) have recently been extended to the vision-language realm, obtaining impressive general multi-modal capabilities. However, the exploration of multi-modal large language models (MLLMs) for remote sensing (RS) data is still in its infancy, and the performance is not satisfactory. In this work, we introduce SkyEyeGPT, a unified multi-modal large language model specifically designed for RS vision-language understanding. To this end, we meticulously curate an RS multi-modal instruction tuning dataset, including single-task and multi-task conversation instructions. After manual verification, we obtain a high-quality RS instruction-following dataset with 968k samples. Our research demonstrates that with a simple yet effective design, SkyEyeGPT works surprisingly well on considerably different tasks without the need for extra encoding modules. Specifically, after projecting RS visual features to the language domain via an alignment layer, they are fed jointly with task-specific instructions into an LLM-based RS decoder to predict answers for RS open-ended tasks. In addition, we design a two-stage tuning method to enhance instruction-following and multi-turn dialogue ability at different granularities. Experiments on 8 datasets for RS vision-language tasks demonstrate SkyEyeGPT's superiority in image-level and region-level tasks, such as captioning and visual grounding. In particular, SkyEyeGPT exhibits encouraging results compared to GPT-4V in some qualitative tests. The online demo, code, and dataset will be released in https://github.com/ZhanYang-nwpu/SkyEyeGPT.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 51 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

33 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  12. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 622
20 more
  1. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  2. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  3. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 339
  4. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 197
  5. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 195
  6. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  7. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  8. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  9. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  10. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  11. 2021Look Before You Leap: Learning Landmark Features for One-Stage Visual Groundingcited by 51
  12. 2022EarthNets: Empowering AI in Earth Observationcited by 40
  13. 2022From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Datacited by 38
  14. 2021Change Detection Meets Visual Question Answeringcited by 29
  15. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
  16. 2023Mono3DVG: 3D Visual Grounding in Monocular Imagescited by 22
  17. 2020ERA: A Dataset and Deep Learning Benchmark for Event Recognition in Aerial Videoscited by 17
  18. 2020CM-Net: Concentric Mask based Arbitrary-Shaped Text Detectioncited by 14
  19. 2022Text Growing on Leafcited by 11
  20. 2023Overcoming Language Bias in Remote Sensing Visual Question Answering via Adversarial Trainingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 49; 2026: 36.

112 papers.

Sort:

By area:

  1. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  2. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  3. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  4. 2024VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysiscited by 84
  5. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  6. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  7. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 55
  8. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  9. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  10. 2024RS-Agent: Automating Remote Sensing Tasks through Intelligent Agentcited by 46
  11. 2024MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generationcited by 41
  12. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation datacited by 39
38 more
  1. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 38
  2. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  4. 2024Large Language Models for Captioning and Retrieving Remote Sensing Imagescited by 30
  5. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  6. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
  7. 2025Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)cited by 28
  8. 2024SARATR-X: Toward Building A Foundation Model for SAR Target Recognitioncited by 25
  9. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  10. 2025RemoteReasoner: Towards Unifying Geospatial Reasoning Workflowcited by 25
  11. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  12. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  13. 2024An LLM Agent for Automatic Geospatial Data Analysiscited by 21
  14. 2025Multi-Agent Geospatial Copilots for Remote Sensing Workflowscited by 20
  15. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 19
  16. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  17. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  18. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  19. 2024RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answeringcited by 15
  20. 2025UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understandingcited by 15
  21. 2025GeoVLM-R1: Reinforcement Fine-Tuning for Improved Remote Sensing Reasoningcited by 15
  22. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  23. 2025LISAT: Language-Instructed Segmentation Assistant for Satellite Imagerycited by 13
  24. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 13
  25. 2025GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenescited by 13
  26. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  27. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  28. 2025GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensingcited by 10
  29. 2025BEDI: A Comprehensive Benchmark for Evaluating Embodied Agents on UAVscited by 10
  30. 2026GeoReason: Aligning Thinking And Answering In Remote Sensing Vision-Language Models Via Logical Consistency Reinforcement Learningcited by 10
  31. 2024REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observationcited by 9
  32. 2026GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagerycited by 9
  33. 2024UniRS: Unifying Multi-temporal Remote Sensing Tasks through Vision Language Modelscited by 8
  34. 2025Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generationcited by 8
  35. 2025BTCChat: Advancing Remote Sensing Bi-temporal Change Captioning with Multimodal Large Language Modelcited by 7
  36. 2025CangLing-KnowFlow: A Unified Knowledge-and-Flow-fused Agent for Comprehensive Remote Sensing Applicationscited by 7
  37. 2026RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMscited by 7
  38. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
  2. 2026SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understanding
  3. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing
  4. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain
  5. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
  6. 2024RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
  7. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing
  8. 2026One Domain, Many Tongues: Composing Domain and Language LoRAs for Cross-Lingual Remote-Sensing MLLMs without Paired Data