mlsanityData to 7 Oct 2026

SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding

arXiv 2406.10100 ↗First posted 14 Jun 2024cs.CV cs.AIcited by 101

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Remote Sensing Large Multi-Modal Models (RSLMMs) are developing rapidly and showcase significant capabilities in remote sensing imagery (RSI) comprehension. However, due to the limitations of existing datasets, RSLMMs have shortcomings in understanding the rich semantic relations among objects in complex remote sensing scenes. To unlock RSLMMs' complex comprehension ability, we propose a large-scale instruction tuning dataset FIT-RS, containing 1,800,851 instruction samples. FIT-RS covers common interpretation tasks and innovatively introduces several complex comprehension tasks of escalating difficulty, ranging from relation reasoning to image-level scene graph generation. Based on FIT-RS, we build the FIT-RSFG benchmark. Furthermore, we establish a new benchmark to evaluate the fine-grained relation comprehension capabilities of LMMs, named FIT-RSRC. Based on combined instruction data, we propose SkySenseGPT, which achieves outstanding performance on both public datasets and FIT-RSFG, surpassing existing RSLMMs. We hope the FIT-RS dataset can enhance the relation comprehension capability of RSLMMs and provide a large-scale fine-grained data source for the remote sensing community. The dataset will be available at https://github.com/Luo-Z13/SkySenseGPT

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 66 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

52 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis29 Mar
  2. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  3. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  4. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  5. 2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing20 Dec
  6. 2023Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment12 Dec
  7. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  8. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  12. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
32 more
  1. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  2. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  3. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  4. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 343
  5. 2022Visual Spatial Reasoningcited by 288
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  7. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  8. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  9. 2021Object Detection in Aerial Images: A Large-Scale Benchmark and Challengescited by 202
  10. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 197
  11. 2020FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understandingcited by 181
  12. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  13. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  14. 2021FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagerycited by 143
  15. 2021Learning High-Precision Bounding Box for Rotated Object Detection via Kullback-Leibler Divergencecited by 130
  16. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  17. 2022Towards Large-Scale Small Object Detection: Survey and Benchmarkscited by 119
  18. 2022MMRotate: A Rotated Object Detection Benchmark using PyTorchcited by 115
  19. 2022Panoptic Scene Graph Generationcited by 114
  20. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  21. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  22. 2021Anchor-free Oriented Proposal Generator for Object Detectioncited by 104
  23. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  24. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  25. 2022Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generationcited by 67
  26. 2023EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answeringcited by 51
  27. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  28. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  29. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39
  30. 2022Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalizationcited by 36
  31. 2024STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagerycited by 31
  32. 2019Relation Network for Multi-label Aerial Image Classificationcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 40; 2026: 47.

101 papers.

Sort:

By area:

  1. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  2. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  3. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  4. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  5. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 55
  6. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  7. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  8. 2025GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolutioncited by 41
  9. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  10. 2025TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysiscited by 35
  11. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  12. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
38 more
  1. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  2. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 19
  3. 2025Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuningcited by 19
  4. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  5. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  6. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  7. 2025Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Modelscited by 17
  8. 2025UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understandingcited by 15
  9. 2025GeoVLM-R1: Reinforcement Fine-Tuning for Improved Remote Sensing Reasoningcited by 15
  10. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 13
  11. 2025GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenescited by 13
  12. 2026OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agentscited by 13
  13. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  14. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMscited by 11
  15. 2025GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensingcited by 10
  16. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
  17. 2025UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenescited by 10
  18. 2026GeoReason: Aligning Thinking And Answering In Remote Sensing Vision-Language Models Via Logical Consistency Reinforcement Learningcited by 10
  19. 2024REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observationcited by 9
  20. 2025A Simple Aerial Detection Baseline of Multimodal Language Modelscited by 9
  21. 2024Enhancing Remote Sensing Vision-Language Models for Zero-Shot Scene Classificationcited by 8
  22. 2024UniRS: Unifying Multi-temporal Remote Sensing Tasks through Vision Language Modelscited by 8
  23. 2025InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognitioncited by 8
  24. 2026RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoningcited by 8
  25. 2025Visual Instruction Pretraining for Domain-Specific Foundation Modelscited by 7
  26. 2026TerraScope: Pixel-Grounded Visual Reasoning for Earth Observationcited by 7
  27. 2026RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMscited by 7
  28. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6
  29. 2024MineAgent: Towards Remote-Sensing Mineral Exploration with Multimodal Large Language Modelscited by 6
  30. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensingcited by 6
  31. 2025Asking like Socrates: Socrates helps VLMs understand remote sensing imagescited by 6
  32. 2025SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Expertscited by 6
  33. 2026VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensingcited by 6
  34. 2026GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensingcited by 6
  35. 2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
  36. 2025Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagerycited by 4
  37. 2025SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Imagescited by 3
  38. 2025Co-Training Vision Language Models for Remote Sensing Multi-task Learningcited by 3

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RS-GPT4V: A Unified Multimodal Instruction-Following Dataset for Remote Sensing Image Understanding
  2. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
  3. 2025Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)
  4. 2026RRS-10K: A Multitask Vision-Language Model Benchmark for Rare Remote Sensing Image Interpretation
  5. 2026Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs
  6. 2024From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
  7. 2025A Vision Centric Remote Sensing Benchmark
  8. 2025TinyRS-R1: Compact Multimodal Language Model for Remote Sensing