# Toward Reliable VLM: A Fine-Grained Benchmark and Framework for Exposure, Bias, and Inference in Korean Street Views

arXiv: https://arxiv.org/abs/2506.03371 · HTML: https://mlsanity.com/paper/2506.03371/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Wide-Area Image Geolocalization with Aerial Reference Imagery](https://mlsanity.com/paper/1510.03743/) (2015)
- [Lending Orientation to Neural Networks for Cross-view Geo-localization](https://mlsanity.com/paper/1903.12351/) (2019)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization](https://mlsanity.com/paper/2002.12186/) (2020)
- [PlaNet - Photo Geolocation with Convolutional Neural Networks](https://mlsanity.com/paper/1602.05314/) (2016)
- [GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition](https://mlsanity.com/paper/2301.02560/) (2023)
- [Private Attribute Inference from Images with Vision-Language Models](https://mlsanity.com/paper/2404.10618/) (2024)
- [Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures](https://mlsanity.com/paper/1601.03896/) (2016)
- [The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks](https://mlsanity.com/paper/2504.15521/) (2025)
- [Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework](https://mlsanity.com/paper/2502.13759/) (2025)
- [Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning](https://mlsanity.com/paper/2307.06166/) (2023)
- [CV-Cities: Advancing Cross-View Geo-Localization in Global Cities](https://mlsanity.com/paper/2411.12431/) (2024)
- [LLMGeo: Benchmarking Large Language Models on Image Geolocation In-the-wild](https://mlsanity.com/paper/2405.20363/) (2024)
- [Towards Interactive Global Geolocation Assistant](https://mlsanity.com/paper/2412.08907/) (2024)
- [Comparing Traditional and LLM-based Search for Image Geolocation](https://mlsanity.com/paper/2401.10184/) (2024)
- [CityGuessr: City-Level Video Geo-Localization on a Global Scale](https://mlsanity.com/paper/2411.06344/) (2024)
- [DeepGeo: Photo Localization with Deep Neural Network](https://mlsanity.com/paper/1810.03077/) (2018)
- [NAVIG: Natural Language-guided Analysis with Vision Language Models for Image Geo-localization](https://mlsanity.com/paper/2502.14638/) (2025)
- [STRIVE: Structured Representation Integrating VLM Reasoning for Efficient Object Navigation](https://mlsanity.com/paper/2505.06729/) (2025)

## Built on by (most cited first, 1 of 1)

- [GeoContext: One Context Ladder, Two Failure Modes in Vision-Language Geolocation: Flat Reliance on User-Provided Location Context and False Confirmation of Location Claims](https://mlsanity.com/paper/2609.05761/) (2026)

## Abstract (arXiv)

Recent advances in vision-language models (VLMs) have enabled accurate image-based geolocation, raising serious concerns about location privacy risks in everyday social media posts. Yet, a systematic evaluation of such risks is still lacking: existing benchmarks show coarse granularity, linguistic bias, and a neglect of multimodal privacy risks. To address these gaps, we introduce KoreaGEO, the first fine-grained, multimodal, and privacy-aware benchmark for geolocation, built on Korean street views. The benchmark covers four socio-spatial clusters and nine place types with rich contextual annotations and two captioning styles that simulate real-world privacy exposure. To evaluate mainstream VLMs, we design a three-path protocol spanning image-only, functional-caption, and high-risk-caption inputs, enabling systematic analysis of localization accuracy, spatial bias, and reasoning behavior. Results show that input modality exerts a stronger influence on localization precision and privacy exposure than model scale or architecture, with high-risk captions substantially boosting accuracy. Moreover, they highlight structural prediction biases toward core cities.