mlsanityData to 7 Oct 2026

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

arXiv 2306.11029 ↗First posted 19 Jun 2023cs.CVcited by 357

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

General-purpose foundation models have led to recent breakthroughs in artificial intelligence. In remote sensing, self-supervised learning (SSL) and Masked Image Modeling (MIM) have been adopted to build foundation models. However, these models primarily learn low-level features and require annotated data for fine-tuning. Moreover, they are inapplicable for retrieval and zero-shot applications due to the lack of language understanding. To address these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. To address the scarcity of pre-training data, we leverage data scaling which converts heterogeneous annotations into a unified image-caption data format based on Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion. By further incorporating UAV imagery, we produce a 12 × larger pretraining dataset than the combination of all available datasets. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, textitk-NN classification, few-shot classification, image-text retrieval, and object counting in remote sensing images. Evaluation on 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, shows that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP beats the state-of-the-art method by 9.14% mean recall on the RSITMD dataset and 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the CLIP baseline by up to 6.39% average accuracy on 12 downstream datasets. Project website: https://github.com/ChenDelong1999/RemoteCLIP

Key facts. Cited by 357 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 139 references that are in the map. First posted on arXiv 19 Jun 2023.

Builds on

83 of 139 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SATIN: A Multi-Task Metadataset for Classifying Satellite Imagery using Vision-Language Models23 Apr

Other references · 82

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  14. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  17. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  18. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 622
  19. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  20. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  21. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  22. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  23. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  24. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  25. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  26. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  27. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 402
  28. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 343
  29. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 339
  30. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  31. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  32. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 302
  33. 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 259
  34. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  35. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 214
  36. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 195
  37. 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 195
  38. 2023Teaching CLIP to Count to Tencited by 194

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 58; 2025: 143; 2026: 144.

357 papers.

Sort:

By area:

  1. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  2. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  3. 2023Foundation Models for Generalist Geospatial Artificial Intelligencecited by 143
  4. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  5. 2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensingcited by 123
  6. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understandingcited by 116
  7. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  8. 2023The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shotcited by 83
  9. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  10. 2024AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalitiescited by 73
  11. 2023Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignmentcited by 71
  12. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
38 more
  1. 2024Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Communitycited by 50
  2. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  3. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  4. 2023A New Learning Paradigm for Foundation Model-based Remote Sensing Change Detectioncited by 47
  5. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 46
  6. 2024MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generationcited by 41
  7. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  8. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 38
  9. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  10. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  11. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  12. 2025TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysiscited by 35
  13. 2024AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generationcited by 32
  14. 2024On the Foundations of Earth and Climate Foundation Modelscited by 31
  15. 2024Large Language Models for Captioning and Retrieving Remote Sensing Imagescited by 30
  16. 2024Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretrainingcited by 27
  17. 2024CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentationcited by 27
  18. 2024CityBench: Evaluating the Capabilities of Large Language Models for Urban Taskscited by 26
  19. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  20. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  21. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  22. 2025FlexiMo: A Flexible Remote Sensing Foundation Modelcited by 24
  23. 2024ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Modelscited by 23
  24. 2023Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learningcited by 22
  25. 2025Multi-Agent Geospatial Copilots for Remote Sensing Workflowscited by 20
  26. 2024CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Modelscited by 19
  27. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 19
  28. 2025Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuningcited by 19
  29. 20243D-GRES: Generalized 3D Referring Expression Segmentationcited by 18
  30. 2024Towards Open-Vocabulary Remote Sensing Image Semantic Segmentationcited by 18
  31. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  32. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  33. 2024Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidancecited by 17
  34. 2025A Comprehensive Survey on Composed Image Retrievalcited by 17
  35. 2025RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Modelscited by 17
  36. 2025RSVLM-QA: A Benchmark Dataset for Remote Sensing Vision Language Model-based Question Answeringcited by 17
  37. 2025RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretationcited by 16
  38. 2025SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understandingcited by 16

50 most cited of 357 All 357 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)
  2. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark
  3. 2025Efficient Few-Shot Learning in Remote Sensing: Fusing Vision and Vision-Language Models
  4. 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
  5. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
  6. 2026SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understanding
  7. 2025Co-Training Vision Language Models for Remote Sensing Multi-task Learning
  8. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model