mlsanityData to 14 Oct 2026

SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing

arXiv 2312.12856 ↗First posted 20 Dec 2023cs.CV cs.AI cs.LGcited by 125

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Remote sensing imagery, despite its broad applications in helping achieve Sustainable Development Goals and tackle climate change, has not yet benefited from the recent advancements of versatile, task-agnostic vision language models (VLMs). A key reason is that the large-scale, semantically diverse image-text dataset required for developing VLMs is still absent for remote sensing images. Unlike natural images, remote sensing images and their associated text descriptions cannot be efficiently collected from the public Internet at scale. In this work, we bridge this gap by using geo-coordinates to automatically connect open, unlabeled remote sensing images with rich semantics covered in OpenStreetMap, and thus construct SkyScript, a comprehensive vision-language dataset for remote sensing images, comprising 2.6 million image-text pairs covering 29K distinct semantic tags. With continual pre-training on this dataset, we obtain a VLM that surpasses baseline models with a 6.2% average accuracy gain in zero-shot scene classification across seven benchmark datasets. It also demonstrates the ability of zero-shot transfer for fine-grained object attribute classification and cross-modal retrieval. We hope this dataset can support the advancement of VLMs for various multi-modal tasks in remote sensing, such as open-vocabulary classification, retrieval, captioning, and text-to-image synthesis.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 45 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

26 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensing19 Jun

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  7. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  8. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
  9. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  10. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  11. 2017Functional Map of the Worldcited by 338
  12. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 308
13 more
  1. 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 261
  2. 2019BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understandingcited by 234
  3. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 215
  4. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 197
  5. 2020Geography-Aware Self-Supervised Learningcited by 190
  6. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  7. 2020Learning Visual Representations with Caption Annotationscited by 123
  8. 2018Tile2Vec: Unsupervised representation learning for spatially distributed datacited by 110
  9. 2016Learning Visual N-Grams from Web Datacited by 95
  10. 2022Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Informationcited by 40
  11. 2022ReforesTree: A Dataset for Estimating Tropical Forest Carbon Stock with Deep Learning and Aerial Imagerycited by 16
  12. 2022Transfer Learning with Pretrained Remote Sensing Transformerscited by 6
  13. 2021GridTracer: Automatic Mapping of Power Grids using Deep Learning and Overhead Imagerycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 25; 2025: 48; 2026: 51.

125 papers.

Sort:

By area:

  1. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 138
  2. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 104
  3. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 85
  4. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 70
  5. 2024Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Communitycited by 50
  6. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 48
  7. 2024MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generationcited by 42
  8. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  9. 2024On the Foundations of Earth and Climate Foundation Modelscited by 33
  10. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  11. 2024GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilotscited by 30
  12. 2024Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretrainingcited by 28
38 more
  1. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  2. 2025TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observationcited by 25
  3. 2024ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Modelscited by 24
  4. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 19
  5. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  6. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 15
  7. 2024Harnessing Massive Satellite Imagery with Efficient Masked Image Modelingcited by 12
  8. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  9. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 11
  10. 2024REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observationcited by 9
  11. 2025OpenRSD: Towards Open-prompts for Object Detection in Remote Sensing Imagescited by 9
  12. 2025AerOSeg: Harnessing SAM for Open-Vocabulary Segmentation in Remote Sensing Imagescited by 9
  13. 2024Enhancing Remote Sensing Vision-Language Models for Zero-Shot Scene Classificationcited by 8
  14. 2025Beyond the Visible: Multispectral Vision-Language Learning for Earth Observationcited by 8
  15. 2025InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognitioncited by 8
  16. 2025SCORE: Scene Context Matters in Open-Vocabulary Remote Sensing Instance Segmentationcited by 8
  17. 2025Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Imagescited by 8
  18. 2025GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Datacited by 8
  19. 2024RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Modelscited by 7
  20. 2024SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level promptingcited by 7
  21. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  22. 2025DOFA-CLIP: Multimodal Vision-Language Foundation Models for Earth Observationcited by 7
  23. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6
  24. 2025Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Modelcited by 6
  25. 2025SOTA: Self-adaptive Optimal Transport for Zero-Shot Classification with Multiple Foundation Modelscited by 6
  26. 2026GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensingcited by 6
  27. 2026Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligencecited by 6
  28. 2023On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcited by 5
  29. 2024A Scoping Review of Earth Observation and Machine Learning for Causal Inference: Implications for the Geography of Povertycited by 5
  30. 2025A large-scale image-text dataset benchmark for farmland segmentationcited by 5
  31. 2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
  32. 2025VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semanticscited by 5
  33. 2026EarthVL: A Progressive Earth Vision-Language Understanding and Generation Frameworkcited by 5
  34. 2026CroBIM-U: Uncertainty-Driven Referring Remote Sensing Image Segmentationcited by 5
  35. 2024PriorCLIP: Visual Prior Guided Vision-Language Model for Remote Sensing Image-Text Retrievalcited by 4
  36. 2024Exploiting Unlabeled Data with Multiple Expert Teachers for Open Vocabulary Aerial Object Detection and Its Orientation Adaptationcited by 4
  37. 2025DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignmentcited by 4
  38. 2025GlobalGeoTree: A Multi-Granular Vision-Language Dataset for Global Tree Species Classificationcited by 4

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations
  2. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
  3. 2025Beyond Pixels: A Training-Free, Text-to-Text Framework for Remote Sensing Image Retrieval
  4. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing
  5. 2026SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understanding
  6. 2026OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs
  7. 2025Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)
  8. 2025Efficient Few-Shot Learning in Remote Sensing: Fusing Vision and Vision-Language Models