# The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework

arXiv: https://arxiv.org/abs/2505.19139 · HTML: https://mlsanity.com/paper/2505.19139/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 58 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 24 of 24)

- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning](https://mlsanity.com/paper/2401.06805/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding](https://mlsanity.com/paper/2105.09996/) (2021)
- [GuardReasoner: Towards Reasoning-based LLM Safeguards](https://mlsanity.com/paper/2501.18492/) (2025)
- [Efficient Inference for Large Reasoning Models: A Survey](https://mlsanity.com/paper/2503.23077/) (2025)
- [SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors](https://mlsanity.com/paper/2403.13438/) (2024)
- [Are Attribute Inference Attacks Just Imputation?](https://mlsanity.com/paper/2209.01292/) (2022)
- [Visual cognition in multimodal large language models](https://mlsanity.com/paper/2311.16093/) (2023)
- [Towards Fine-Grained Reasoning for Fake News Detection](https://mlsanity.com/paper/2110.15064/) (2021)
- [On the (In)Feasibility of Attribute Inference Attacks on Machine Learning Models](https://mlsanity.com/paper/2103.07101/) (2021)
- [SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models](https://mlsanity.com/paper/2504.08813/) (2025)
- [Attribute Recognition by Joint Recurrent Learning of Context and Correlation](https://mlsanity.com/paper/1709.08553/) (2017)
- [Fairness in Clustering with Multiple Sensitive Attributes](https://mlsanity.com/paper/1910.05113/) (2019)
- [Attribute Privacy: Framework and Mechanisms](https://mlsanity.com/paper/2009.04013/) (2020)
- [A Survey of Privacy Vulnerabilities of Mobile Device Sensors](https://mlsanity.com/paper/2106.10154/) (2021)
- [A Survey on Safe Multi-Modal Learning System](https://mlsanity.com/paper/2402.05355/) (2024)
- [TUNI: A Textual Unimodal Detector for Identity Inference in CLIP Models](https://mlsanity.com/paper/2405.14517/) (2024)
- [DAA: A Delta Age AdaIN operation for age estimation via binary code transformer](https://mlsanity.com/paper/2303.07929/) (2023)
- [Security and Privacy Problems in Voice Assistant Applications: A Survey](https://mlsanity.com/paper/2304.09486/) (2023)
- [Consensus of Hybrid Multi-agent Systems](https://mlsanity.com/paper/1512.03189/) (2015)

## Built on by (most cited first, 9 of 9)

- [A Vision for Access Control in LLM-based Agent Systems](https://mlsanity.com/paper/2510.11108/) (2025)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attacks](https://mlsanity.com/paper/2512.18264/) (2025)
- [Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs](https://mlsanity.com/paper/2603.21573/) (2026)
- [What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media](https://mlsanity.com/paper/2606.06784/) (2026)
- [When Ads Become Profiles: Uncovering the Invisible Risk of Web Advertising at Scale with LLMs](https://mlsanity.com/paper/2509.18874/) (2025)
- [How Implicit Bias Accumulates and Propagates in LLM Long-term Memory](https://mlsanity.com/paper/2602.01558/) (2026)
- [Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation](https://mlsanity.com/paper/2604.09368/) (2026)
- [AccretionLink: On-Device Auditing of Exposure-Control Attacks on Attribute Inference](https://mlsanity.com/paper/2608.14735/) (2026)

## Abstract (arXiv)

Our research reveals a new privacy risk associated with the vision-language model (VLM) agentic framework: the ability to infer sensitive attributes (e.g., age and health information) and even abstract ones (e.g., personality and social traits) from a set of personal images, which we term "image private attribute profiling." This threat is particularly severe given that modern apps can easily access users' photo albums, and inference from image sets enables models to exploit inter-image relations for more sophisticated profiling. However, two main challenges hinder our understanding of how well VLMs can profile an individual from a few personal photos: (1) the lack of benchmark datasets with multi-image annotations for private attributes, and (2) the limited ability of current multimodal large language models (MLLMs) to infer abstract attributes from large image collections. In this work, we construct PAPI, the largest dataset for studying private attribute profiling in personal images, comprising 2,510 images from 251 individuals with 3,012 annotated privacy attributes. We also propose HolmesEye, a hybrid agentic framework that combines VLMs and LLMs to enhance privacy inference. HolmesEye uses VLMs to extract both intra-image and inter-image information and LLMs to guide the inference process as well as consolidate the results through forensic analysis, overcoming existing limitations in long-context visual reasoning. Experiments reveal that HolmesEye achieves a 10.8% improvement in average accuracy over state-of-the-art baselines and surpasses human-level performance by 15.0% in predicting abstract attributes. This work highlights the urgency of addressing privacy risks in image-based profiling and offers both a new dataset and an advanced framework to guide future research in this area.