# Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images

arXiv: https://arxiv.org/abs/1703.10660 · HTML: https://mlsanity.com/paper/1703.10660/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2017.

## Built on by (most cited first, 30 of 101)

- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning](https://mlsanity.com/paper/2309.06597/) (2023)
- [SPAct: Self-supervised Privacy Preservation for Action Recognition](https://mlsanity.com/paper/2203.15205/) (2022)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech](https://mlsanity.com/paper/2106.01625/) (2021)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Gradient Obfuscation Gives a False Sense of Security in Federated Learning](https://mlsanity.com/paper/2206.04055/) (2022)
- [Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression](https://mlsanity.com/paper/2301.13838/) (2023)
- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation](https://mlsanity.com/paper/2103.05271/) (2021)
- [HowToCaption: Prompting LLMs to Transform Video Annotations at Scale](https://mlsanity.com/paper/2310.04900/) (2023)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2211.15103/) (2022)
- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [TeD-SPAD: Temporal Distinctiveness for Self-supervised Privacy-preservation for video Anomaly Detection](https://mlsanity.com/paper/2308.11072/) (2023)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [IC3: Image Captioning by Committee Consensus](https://mlsanity.com/paper/2302.01328/) (2023)
- [VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2206.12972/) (2022)
- [Affection: Learning Affective Explanations for Real-World Visual Data](https://mlsanity.com/paper/2210.01946/) (2022)
- [PrivHAR: Recognizing Human Actions From Privacy-preserving Lens](https://mlsanity.com/paper/2206.03891/) (2022)
- [Privacy-Preserving Action Recognition via Motion Difference Quantization](https://mlsanity.com/paper/2208.02459/) (2022)
- [Learning Distinct and Representative Styles for Image Captioning](https://mlsanity.com/paper/2209.08231/) (2022)
- [Privacy-Aware Visual Language Models](https://mlsanity.com/paper/2405.17423/) (2024)
- [CapOnImage: Context-driven Dense-Captioning on Image](https://mlsanity.com/paper/2204.12974/) (2022)
- [Simple Token-Level Confidence Improves Caption Correctness](https://mlsanity.com/paper/2305.07021/) (2023)
- [Going Grayscale: The Road to Understanding and Improving Unlearnable Examples](https://mlsanity.com/paper/2111.13244/) (2021)
- [Show, Interpret and Tell: Entity-aware Contextualised Image Captioning in Wikipedia](https://mlsanity.com/paper/2209.10474/) (2022)

## Abstract (arXiv)

With an increasing number of users sharing information online, privacy implications entailing such actions are a major concern. For explicit content, such as user profile or GPS data, devices (e.g. mobile phones) as well as web services (e.g. Facebook) offer to set privacy settings in order to enforce the users' privacy preferences. We propose the first approach that extends this concept to image content in the spirit of a Visual Privacy Advisor. First, we categorize personal information in images into 68 image attributes and collect a dataset, which allows us to train models that predict such information directly from images. Second, we run a user study to understand the privacy preferences of different users w.r.t. such attributes. Third, we propose models that predict user specific privacy score from images in order to enforce the users' privacy preferences. Our model is trained to predict the user specific privacy risk and even outperforms the judgment of the users, who often fail to follow their own privacy preferences on image data.