mlsanityData to 14 Oct 2026

Private Attribute Inference from Images with Vision-Language Models

arXiv 2404.10618 ↗First posted 16 Apr 2024cs.AI cs.CV cs.LGcited by 37

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

As large language models (LLMs) become ubiquitous in our daily tasks and digital interactions, associated privacy risks are increasingly in focus. While LLM privacy research has primarily focused on the leakage of model training data, it has recently been shown that LLMs can make accurate privacy-infringing inferences from previously unseen texts. With the rise of vision-language models (VLMs), capable of understanding both images and text, a key question is whether this concern transfers to the previously unexplored domain of benign images posted online. To answer this question, we compile an image dataset with human-annotated labels of the image owner's personal attributes. In order to understand the privacy risks posed by VLMs beyond traditional human attribute recognition, our dataset consists of images where the inferable private attributes do not stem from direct depictions of humans. On this dataset, we evaluate 7 state-of-the-art VLMs, finding that they can infer various personal attributes at up to 77.6% accuracy. Concerningly, we observe that accuracy scales with the general capabilities of the models, implying that future models can be misused as stronger inferential adversaries, establishing an imperative for the development of adequate defenses.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 53 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

29 of 53 references.

References · 29

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2021Ethical and social risks of harm from Language Modelscited by 966
17 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  2. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  8. 2023Multimodal Large Language Models: A Surveycited by 195
  9. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 175
  10. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 129
  11. 2022Language Models are General-Purpose Interfacescited by 97
  12. 2017HydraPlus-Net: Attentive Deep Features for Pedestrian Analysiscited by 89
  13. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  14. 2019Pedestrian Attribute Recognition: A Surveycited by 50
  15. 2023Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusioncited by 24
  16. 2024Large Language Models are Advanced Anonymizerscited by 22
  17. 2021A Review on Visual Privacy Preservation Techniques for Active and Assisted Livingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 20.

37 papers.

Sort:

By area:

  1. 2024A Synthetic Dataset for Personal Attribute Inferencecited by 36
  2. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  3. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoderscited by 13
  4. 2024Privacy-Aware Visual Language Modelscited by 12
  5. 2025Evaluating Precise Geolocation Inference Capabilities of Vision Language Modelscited by 12
  6. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulatedcited by 12
  7. 2025Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safetycited by 7
  8. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  9. 2025LLMs unlock new paths to monetizing exploitscited by 5
  10. 2024Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Modelscited by 3
  11. 2025Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attackscited by 3
  12. 2025Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inferencecited by 2
25 more
  1. 2026Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMscited by 2
  2. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  3. 2026What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Mediacited by 2
  4. 2025Toward Reliable VLM: A Fine-Grained Benchmark and Framework for Exposure, Bias, and Inference in Korean Street Viewscited by 1
  5. 2025Through Their Eyes: User Perceptions on Sensitive Attribute Inference of Social Media Videos by Visual Language Modelscited by 1
  6. 2025When Ads Become Profiles: Uncovering the Invisible Risk of Web Advertising at Scale with LLMscited by 1
  7. 2025GEO-Detective: Unveiling Location Privacy Risks in Images with LLM Agentscited by 1
  8. 2026Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?cited by 1
  9. 2026Differentially Private Multimodal In-Context Learningcited by 1
  10. 2026Towards Context-Aware Image Anonymization with Multi-Agent Reasoningcited by 1
  11. 2026Unsafe2Safe: Controllable Image Anonymization for Downstream Utilitycited by 1
  12. 2026Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mindcited by 1
  13. 2026CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agentscited by 1
  14. 2026MemLeak: Diagnosing Information Leaks in Multimodal Agent Memorycited by 1
  15. 2026Defending Wearable VLMs Against Private Attribute Inferencecited by 1
  16. 2025Rethinking Post-Unlearning Behavior of Large Vision-Language Modelscited by 0
  17. 2026Which private attributes do VLMs agree on and predict well?cited by 0
  18. 2026Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gatingcited by 0
  19. 2026Assessing Privacy Preservation and Utility in Online Vision-Language Modelscited by 0
  20. 2026Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agentscited by 0
  21. 2026Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMscited by 0
  22. 20263D FaceShell: Attribute Transfer in 3D Face Avatars as a VLM Defense Mechanismcited by 0
  23. 2026"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverablescited by 0
  24. 2026SAGE: Efficient Protected Vision-Language Inference on Smart Glassescited by 0
  25. 2026COMPASS: Membership Inference Attack Against Vision Large Language Models via Adaptive Noise Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models
  2. 2024The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models
  3. 2025The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework
  4. 2025The Pervasive Blind Spot: Benchmarking VLM Inference Risks on Everyday Personal Videos
  5. 2026DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models
  6. 2025On the Robustness of Vision-Language Models in Zero-shot Privacy Classification
  7. 2025Assessing Visual Privacy Risks in Multimodal AI: A Novel Taxonomy-Grounded Evaluation of Vision-Language Models
  8. 2026Vision Language Model Helps Private Information De-Identification in Vision Data