# Learning to Evaluate Performance of Multi-modal Semantic Localization arXiv: https://arxiv.org/abs/2209.06515 ยท HTML: https://mlsanity.com/paper/2209.06515/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 18 of its 58 references that are in the map. - First posted on arXiv 14 Sep 2022. ## Builds on (most cited first, 18 of 18) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021) - [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021) - [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018) - [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018) - [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017) - [CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval](https://mlsanity.com/paper/1909.05506/) (2019) - [Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval](https://mlsanity.com/paper/2204.09868/) (2022) - [Fusion of Detected Objects in Text for Visual Question Answering](https://mlsanity.com/paper/1908.05054/) (2019) - [Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information](https://mlsanity.com/paper/2204.09860/) (2022) - [A Real-time Global Inference Network for One-stage Referring Expression Comprehension](https://mlsanity.com/paper/1912.03478/) (2019) - [Relationship-Embedded Representation Learning for Grounding Referring Expressions](https://mlsanity.com/paper/1906.04464/) (2019) - [A General Gaussian Heatmap Label Assignment for Arbitrary-Oriented Object Detection](https://mlsanity.com/paper/2109.12848/) (2021) - [Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking](https://mlsanity.com/paper/1908.04011/) (2019) - [Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval](https://mlsanity.com/paper/1610.03023/) (2016) - [Exploiting Deep Features for Remote Sensing Image Retrieval: A Systematic Investigation](https://mlsanity.com/paper/1707.07321/) (2017) - [Unsupervised Contrastive Hashing for Cross-Modal Retrieval in Remote Sensing](https://mlsanity.com/paper/2204.08707/) (2022) ## Built on by (most cited first, 12 of 12) - [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023) - [Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community](https://mlsanity.com/paper/2408.09110/) (2024) - [Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrieval](https://mlsanity.com/paper/2308.12509/) (2023) - [Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval](https://mlsanity.com/paper/2310.08276/) (2023) - [DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignment](https://mlsanity.com/paper/2503.19311/) (2025) - [Semantic-Aware Ship Detection with Vision-Language Integration](https://mlsanity.com/paper/2508.15930/) (2025) - [Open-Text Aerial Detection: A Unified Framework For Aerial Visual Grounding And Detection](https://mlsanity.com/paper/2602.07827/) (2026) - [Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations](https://mlsanity.com/paper/2409.07048/) (2024) - [SARVLM: A Vision Language Foundation Model for Semantic Understanding in SAR Imagery](https://mlsanity.com/paper/2510.22665/) (2025) - [FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understanding](https://mlsanity.com/paper/2511.14901/) (2025) - [XeMap: Contextual Referring in Large-Scale Remote Sensing Environments](https://mlsanity.com/paper/2505.00738/) (2025) - [Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data](https://mlsanity.com/paper/2609.03391/) (2026) ## Abstract (arXiv) Semantic localization (SeLo) refers to the task of obtaining the most relevant locations in large-scale remote sensing (RS) images using semantic information such as text. As an emerging task based on cross-modal retrieval, SeLo achieves semantic-level retrieval with only caption-level annotation, which demonstrates its great potential in unifying downstream tasks. Although SeLo has been carried out successively, but there is currently no work has systematically explores and analyzes this urgent direction. In this paper, we thoroughly study this field and provide a complete benchmark in terms of metrics and testdata to advance the SeLo task. Firstly, based on the characteristics of this task, we propose multiple discriminative evaluation metrics to quantify the performance of the SeLo task. The devised significant area proportion, attention shift distance, and discrete attention distance are utilized to evaluate the generated SeLo map from pixel-level and region-level. Next, to provide standard evaluation data for the SeLo task, we contribute a diverse, multi-semantic, multi-objective Semantic Localization Testset (AIR-SLT). AIR-SLT consists of 22 large-scale RS images and 59 test cases with different semantics, which aims to provide a comprehensive evaluations for retrieval models. Finally, we analyze the SeLo performance of RS cross-modal retrieval models in detail, explore the impact of different variables on this task, and provide a complete benchmark for the SeLo task. We have also established a new paradigm for RS referring expression comprehension, and demonstrated the great advantage of SeLo in semantics through combining it with tasks such as detection and road extraction. The proposed evaluation metrics, semantic localization testsets, and corresponding scripts have been open to access at github.com/xiaoyuan1996/SemanticLocalizationMetrics .