# Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images

arXiv: https://arxiv.org/abs/2307.15904 · HTML: https://mlsanity.com/paper/2307.15904/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 29 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [Fine-grained Image Captioning with CLIP Reward](https://mlsanity.com/paper/2205.13115/) (2022)
- [CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP](https://mlsanity.com/paper/2203.00386/) (2022)
- [Extending CLIP for Category-to-image Retrieval in E-commerce](https://mlsanity.com/paper/2112.11294/) (2021)
- [Learning a Dynamic Map of Visual Appearance](https://mlsanity.com/paper/2012.14885/) (2020)
- [Understanding and Mapping Natural Beauty](https://mlsanity.com/paper/1612.03142/) (2016)
- [Beyond Spatial Auto-Regressive Models: Predicting Housing Prices with Satellite Imagery](https://mlsanity.com/paper/1610.04805/) (2016)
- [Dynamic Traffic Modeling From Overhead Imagery](https://mlsanity.com/paper/2012.10530/) (2020)
- [DeepDPM: Dynamic Population Mapping via Deep Neural Network](https://mlsanity.com/paper/1811.02644/) (2018)
- [What Goes Where: Predicting Object Distributions from Above](https://mlsanity.com/paper/1808.00995/) (2018)

## Built on by (most cited first, 19 of 19)

- [Mission Critical -- Satellite Data is a Distinct Modality in Machine Learning](https://mlsanity.com/paper/2402.01444/) (2024)
- [Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching](https://mlsanity.com/paper/2311.12751/) (2023)
- [TaxaBind: A Unified Embedding Space for Ecological Applications](https://mlsanity.com/paper/2411.00683/) (2024)
- [GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesis](https://mlsanity.com/paper/2404.06637/) (2024)
- [RANGE: Retrieval Augmented Neural Fields for Multi-Resolution Geo-Embeddings](https://mlsanity.com/paper/2502.19781/) (2025)
- [Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment](https://mlsanity.com/paper/2402.09816/) (2024)
- [BirdSAT: Cross-View Contrastive Masked Autoencoders for Bird Species Classification and Mapping](https://mlsanity.com/paper/2310.19168/) (2023)
- [GOMAA-Geo: GOal Modality Agnostic Active Geo-localization](https://mlsanity.com/paper/2406.01917/) (2024)
- [GEOBIND: Binding Text, Image, and Audio through Satellite Images](https://mlsanity.com/paper/2404.11720/) (2024)
- [SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level prompting](https://mlsanity.com/paper/2412.08536/) (2024)
- [VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semantics](https://mlsanity.com/paper/2511.07744/) (2025)
- [EcoWikiRS: Learning Ecological Representation of Satellite Images from Weak Supervision with Species Observations and Wikipedia](https://mlsanity.com/paper/2504.19742/) (2025)
- [GeoExplorer: Active Geo-localization with Curiosity-Driven Exploration](https://mlsanity.com/paper/2508.00152/) (2025)
- [Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty?](https://mlsanity.com/paper/2508.01109/) (2025)
- [TimeSenCLIP: A Time Series Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2508.11919/) (2025)
- [ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology](https://mlsanity.com/paper/2511.02946/) (2025)
- [Few-Shot Remote Sensing Image Scene Classification with CLIP and Prompt Learning](https://mlsanity.com/paper/2510.24321/) (2025)
- [BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment](https://mlsanity.com/paper/2603.23883/) (2026)
- [SatBLIP: Context Understanding and Feature Identification from Satellite Imagery with Vision-Language Learning](https://mlsanity.com/paper/2604.14373/) (2026)

## Abstract (arXiv)

We propose a weakly supervised approach for creating maps using free-form textual descriptions. We refer to this work of creating textual maps as zero-shot mapping. Prior works have approached mapping tasks by developing models that predict a fixed set of attributes using overhead imagery. However, these models are very restrictive as they can only solve highly specific tasks for which they were trained. Mapping text, on the other hand, allows us to solve a large variety of mapping problems with minimal restrictions. To achieve this, we train a contrastive learning framework called Sat2Cap on a new large-scale dataset with 6.1M pairs of overhead and ground-level images. For a given location and overhead image, our model predicts the expected CLIP embeddings of the ground-level scenery. The predicted CLIP embeddings are then used to learn about the textual space associated with that location. Sat2Cap is also conditioned on date-time information, allowing it to model temporally varying concepts over a location. Our experimental results demonstrate that our models successfully capture ground-level concepts and allow large-scale mapping of fine-grained textual queries. Our approach does not require any text-labeled data, making the training easily scalable. The code, dataset, and models will be made publicly available.