mlsanityData to 7 Oct 2026

Granular Privacy Control for Geolocation with Vision Language Models

arXiv 2407.04952 ↗First posted 6 Jul 2024cs.CL cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision Language Models (VLMs) are rapidly advancing in their capability to answer information-seeking questions. As these models are widely deployed in consumer applications, they could lead to new privacy risks due to emergent abilities to identify people in photos, geolocate images, etc. As we demonstrate, somewhat surprisingly, current open-source and proprietary VLMs are very capable image geolocators, making widespread geolocation with VLMs an immediate privacy risk, rather than merely a theoretical future concern. As a first step to address this challenge, we develop a new benchmark, GPTGeoChat, to test the ability of VLMs to moderate geolocation dialogues with users. We collect a set of 1,000 image geolocation conversations between in-house annotators and GPT-4v, which are annotated with the granularity of location information revealed at each turn. Using this new dataset, we evaluate the ability of various VLMs to moderate GPT-4v geolocation conversations by determining when too much location information has been revealed. We find that custom fine-tuned models perform on par with prompted API-based models when identifying leaked location information at the country or city level; however, fine-tuning on supervised data appears to be needed to accurately moderate finer granularities, such as the name of a restaurant or building.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 56 references that are in the map. First posted on arXiv 6 Jul 2024.

Builds on

31 of 56 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  12. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
19 more
  1. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  2. 2023GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localizationcited by 185
  3. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  4. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  5. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  6. 2016PlaNet - Photo Geolocation with Convolutional Neural Networkscited by 114
  7. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  8. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  9. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  10. 2017Revisiting IM2GPS in the Deep Learning Eracited by 79
  11. 2021Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contextscited by 56
  12. 2023Where We Are and What We're Looking At: Query Based Worldwide Image Geo-localization Using Hierarchies and Scenescited by 55
  13. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  14. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  15. 2024G3: An Effective and Adaptive Framework for Worldwide Geolocalization Using Large Multi-Modality Modelscited by 33
  16. 2023Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Riskscited by 31
  17. 2024Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generationcited by 31
  18. 2022Geographic Adaptation of Pretrained Language Modelscited by 14
  19. 2017A Neural Model for User Geolocation and Lexical Dialectologycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 7.

19 papers.

Sort:

By area:

  1. 2024Image-Based Geolocation Using Large Vision-Language Modelscited by 17
  2. 2024Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbotscited by 16
  3. 2025Position: Privacy Is Not Just Memorization!cited by 15
  4. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  5. 2024Have Large Vision-Language Models Mastered Art History?cited by 9
  6. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scalescited by 5
  7. 2025Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?cited by 3
  8. 2025Evaluation of Geolocation Capabilities of Multimodal Large Language Models and Analysis of Associated Privacy Riskscited by 3
  9. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 3
  10. 2025VLM-Guided Visual Place Recognition for Planet-Scale Geo-Localizationcited by 2
  11. 2025Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attackscited by 2
  12. 2026Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?cited by 1
7 more
  1. 2026Differentially Private Multimodal In-Context Learningcited by 1
  2. 2025GEO-Detective: Unveiling Location Privacy Risks in Images with LLM Agentscited by 0
  3. 2026GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Predictioncited by 0
  4. 2026LocationAgent: A Hierarchical Agent for Image Geolocation via Decoupling Strategy and Evidence from Parametric Knowledgecited by 0
  5. 2026Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMscited by 0
  6. 2026Defending Wearable VLMs Against Private Attribute Inferencecited by 0
  7. 2026"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverablescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Precise Geolocation Inference Capabilities of Vision Language Models
  2. 2025Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models
  3. 2023GeoLocator: a location-integrated large multimodal model for inferring geo-privacy
  4. 2026Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models
  5. 2025Toward Reliable VLM: A Fine-Grained Benchmark and Framework for Exposure, Bias, and Inference in Korean Street Views
  6. 2025Beyond Pixels: Semantic-aware Typographic Attack for Geo-Privacy Protection
  7. 2026GeoRC: A Benchmark for Geolocation Reasoning Chains
  8. 2026GeoContext: One Context Ladder, Two Failure Modes in Vision-Language Geolocation: Flat Reliance on User-Provided Location Context and False Confirmation of Location Claims