mlsanityData to 7 Oct 2026

Which private attributes do VLMs agree on and predict well?

arXiv 2602.07931 ↗First posted 8 Feb 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Visual Language Models (VLMs) are often used for zero-shot detection of visual attributes in the image. We present a zero-shot evaluation of open-source VLMs for privacy-related attribute recognition. We identify the attributes for which VLMs exhibit strong inter-annotator agreement, and discuss the disagreement cases of human and VLM annotations. Our results show that when evaluated against human annotations, VLMs tend to predict the presence of privacy attributes more often than human annotators. In addition to this, we find that in cases of high inter-annotator agreement between VLMs, they can complement human annotation by identifying attributes overlooked by human annotators. This highlights the potential of VLMs to support privacy annotations in large-scale image datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 20 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

14 of 20 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Models27 Dec
  2. 2024Private Attribute Inference from Images with Vision-Language Models16 Apr
  3. 2024Red Teaming Visual Language Models23 Jan

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2017Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Imagescited by 101
  7. 2025PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliancecited by 34
  8. 2025MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Modelscited by 14
  9. 2023GeoLocator: a location-integrated large multimodal model for inferring geo-privacycited by 13
  10. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  11. 2024Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Datasetcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attacks
  2. 2025On the Robustness of Vision-Language Models in Zero-shot Privacy Classification
  3. 2024Privacy-Aware Visual Language Models
  4. 2024The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models
  5. 2025Through Their Eyes: User Perceptions on Sensitive Attribute Inference of Social Media Videos by Visual Language Models
  6. 2025MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models
  7. 2026Defending Wearable VLMs Against Private Attribute Inference
  8. 2026DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models