# Image-guided topic modeling for interpretable privacy classification

arXiv: https://arxiv.org/abs/2409.18674 · HTML: https://mlsanity.com/paper/2409.18674/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 76 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Parametric UMAP embeddings for representation and semi-supervised learning](https://mlsanity.com/paper/2009.12981/) (2020)
- [Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images](https://mlsanity.com/paper/1703.10660/) (2017)
- [Interpretability Beyond Classification Output: Semantic Bottleneck Networks](https://mlsanity.com/paper/1907.10882/) (2019)
- [Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models](https://mlsanity.com/paper/2310.03182/) (2023)
- [Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?](https://mlsanity.com/paper/2203.17271/) (2022)
- [Image Privacy Prediction Using Deep Neural Networks](https://mlsanity.com/paper/1903.03695/) (2019)
- [Zero-Bias Deep Learning for Accurate Identification of Internet of Things (IoT) Devices](https://mlsanity.com/paper/2009.02267/) (2020)
- [Dynamic Deep Multi-modal Fusion for Image Privacy Prediction](https://mlsanity.com/paper/1902.10796/) (2019)
- [DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection](https://mlsanity.com/paper/2203.09121/) (2022)
- [Content-based Graph Privacy Advisor](https://mlsanity.com/paper/2210.11169/) (2022)
- [Human-interpretable and deep features for image privacy classification](https://mlsanity.com/paper/2310.19582/) (2023)

## Built on by (most cited first, 4 of 4)

- [On the Robustness of Vision-Language Models in Zero-shot Privacy Classification](https://mlsanity.com/paper/2510.09253/) (2025)
- [The impact of abstract and object tags on image privacy classification](https://mlsanity.com/paper/2510.07976/) (2025)
- [Cross-modal Counterfactual Explanations: Uncovering Decision Factors and Dataset Biases in Subjective Classification](https://mlsanity.com/paper/2512.18864/) (2025)
- [PrivLEX: Detecting legal concepts in images through Vision-Language Models](https://mlsanity.com/paper/2601.09449/) (2026)

## Abstract (arXiv)

Predicting and explaining the private information contained in an image in human-understandable terms is a complex and contextual task. This task is challenging even for large language models. To facilitate the understanding of privacy decisions, we propose to predict image privacy based on a set of natural language content descriptors. These content descriptors are associated with privacy scores that reflect how people perceive image content. We generate descriptors with our novel Image-guided Topic Modeling (ITM) approach. ITM leverages, via multimodality alignment, both vision information and image textual descriptions from a vision language model. We use the ITM-generated descriptors to learn a privacy predictor, Priv×ITM, whose decisions are interpretable by design. Our Priv×ITM classifier outperforms the reference interpretable method by 5 percentage points in accuracy and performs comparably to the current non-interpretable state-of-the-art model.