# On the Robustness of Vision-Language Models in Zero-shot Privacy Classification

arXiv: https://arxiv.org/abs/2510.09253 · HTML: https://mlsanity.com/paper/2510.09253/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 18 of 18)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images](https://mlsanity.com/paper/1703.10660/) (2017)
- [A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models](https://mlsanity.com/paper/2110.08484/) (2021)
- [MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models](https://mlsanity.com/paper/2406.07057/) (2024)
- [Image Privacy Prediction Using Deep Neural Networks](https://mlsanity.com/paper/1903.03695/) (2019)
- [Dynamic Deep Multi-modal Fusion for Image Privacy Prediction](https://mlsanity.com/paper/1902.10796/) (2019)
- [Assessing Visual Privacy Risks in Multimodal AI: A Novel Taxonomy-Grounded Evaluation of Vision-Language Models](https://mlsanity.com/paper/2509.23827/) (2025)
- [Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Dataset](https://mlsanity.com/paper/2410.09416/) (2024)
- [REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Models](https://mlsanity.com/paper/2503.16566/) (2025)
- [DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection](https://mlsanity.com/paper/2203.09121/) (2022)
- [Image-guided topic modeling for interpretable privacy classification](https://mlsanity.com/paper/2409.18674/) (2024)
- ['Surprised, Shocked, Worried': User Reactions to Facebook Data Collection from Third Parties](https://mlsanity.com/paper/2209.08048/) (2022)
- [Learning Privacy from Visual Entities](https://mlsanity.com/paper/2503.12464/) (2025)

## Built on by (most cited first, 1 of 1)

- [PrivLEX: Detecting legal concepts in images through Vision-Language Models](https://mlsanity.com/paper/2601.09449/) (2026)

## Abstract (arXiv)

Automatic systems for document understanding require multimodal models that accurately identify sensitive visual content, even in the presence of image degradations. Instruction-following large Vision-Language Models (VLMs) are expected to generalise across domains and tasks without requiring any specific adaptation. In this work, we systematically analyse whether VLMs can be used reliably for image privacy classification in a zero-shot setup. We evaluate and compare the classification performance of three open-source VLMs against purposely built models on two public, standard benchmarks. We assess robustness to image degradations caused by perturbations such as compression, light variations, and random noise, and analyse inference speed and parameter count to deploy VLMs in privacy-aware document processing pipelines. Our results show that large VLMs are robust to input perturbations but are less accurate (and much slower) than smaller privacy models. Scaling alone is not sufficient, highlighting the advantages of models specifically designed for privacy classification.