# Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?

arXiv: https://arxiv.org/abs/2501.16947 · HTML: https://mlsanity.com/paper/2501.16947/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval](https://mlsanity.com/paper/2011.12172/) (2020)
- [Wide-Area Image Geolocalization with Aerial Reference Imagery](https://mlsanity.com/paper/1510.03743/) (2015)
- [TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization](https://mlsanity.com/paper/2204.00097/) (2022)
- [Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching](https://mlsanity.com/paper/2005.03860/) (2020)
- [Revisiting IM2GPS in the Deep Learning Era](https://mlsanity.com/paper/1705.04838/) (2017)
- [Cross-View Image Matching for Geo-localization in Urban Environments](https://mlsanity.com/paper/1703.07815/) (2017)
- [CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps](https://mlsanity.com/paper/1808.02130/) (2018)
- [Where in the World is this Image? Transformer-based Geo-localization in the Wild](https://mlsanity.com/paper/2204.13861/) (2022)
- [PIGEON: Predicting Image Geolocations](https://mlsanity.com/paper/2307.05845/) (2023)
- [CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation](https://mlsanity.com/paper/2211.16649/) (2022)
- [Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation](https://mlsanity.com/paper/2403.19584/) (2024)
- [Granular Privacy Control for Geolocation with Vision Language Models](https://mlsanity.com/paper/2407.04952/) (2024)
- [Leveraging EfficientNet and Contrastive Learning for Accurate Global-scale Location Estimation](https://mlsanity.com/paper/2105.07645/) (2021)
- [G^3: Geolocation via Guidebook Grounding](https://mlsanity.com/paper/2211.15521/) (2022)
- [Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework](https://mlsanity.com/paper/2408.11312/) (2024)
- [Online Visual Place Recognition via Saliency Re-identification](https://mlsanity.com/paper/2007.14549/) (2020)
- [On Motion Blur and Deblurring in Visual Place Recognition](https://mlsanity.com/paper/2412.07751/) (2024)

## Built on by (most cited first, 3 of 3)

- [VLM-Guided Visual Place Recognition for Planet-Scale Geo-Localization](https://mlsanity.com/paper/2507.17455/) (2025)
- [MinkUNeXt-SI: Improving point cloud-based place recognition including spherical coordinates and LiDAR intensity](https://mlsanity.com/paper/2505.17591/) (2025)
- [Breaking D'ej`a Vu: Independent Auditing of Visual Place Recognition through Vision-Language Reasoning](https://mlsanity.com/paper/2607.12818/) (2026)

## Abstract (arXiv)

The advances in Vision-Language models (VLMs) offer exciting opportunities for robotic applications involving image geo-localization - the problem of identifying the geo-coordinates of a place based on visual data only. In robotics, such capabilities are particularly relevant to the global re-localization stage of the kidnapped robot problem, where a robot must recover its pose without prior knowledge of its location. Recent work has focused on using a VLM as embedding extractor for geo-localization. However, the most sophisticated VLMs may only be available as black boxes that are accessible through an API, and come with a number of limitations: there is no access to training data, model features and gradients; retraining is not possible; and the number of predictions may be limited by the API. The potential of state-of-the-art VLMs as a stand-alone, zero-shot geo-localization systems at planet scale using a single text-based prompt is largely unexplored. To bridge this gap, this paper undertakes the first systematic study, to the best of our knowledge, to investigate state-of-the-art generative VLMs as stand-alone, zero-shot geo-localization systems in a black-box setting with realistic constraints. We consider three main scenarios for this thorough investigation: a) fixed text-based prompt; b) semantically-equivalent text-based prompts; and c) semantically-equivalent query images. Beyond standard accuracy, we introduce model consistency as a metric to account for the auto-regressive and probabilistic nature of generative VLMs. Our findings reveal that while VLMs demonstrate strong coarse-level localization and navigation priors, fine-grained localization degrades significantly under realistic variations, highlighting reliability challenges for deploying generative VLMs in robust, open-world robotic navigation systems.