# Does CLIP Know My Face?

arXiv: https://arxiv.org/abs/2209.07341 · HTML: https://mlsanity.com/paper/2209.07341/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 95 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [Membership Leakage in Label-Only Exposures](https://mlsanity.com/paper/2007.15528/) (2020)
- [Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?](https://mlsanity.com/paper/2202.06675/) (2022)
- [MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning](https://mlsanity.com/paper/2112.05253/) (2021)

## Built on by (most cited first, 19 of 19)

- [Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attacks](https://mlsanity.com/paper/2310.06549/) (2023)
- [Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics](https://mlsanity.com/paper/2402.09225/) (2024)
- [Gibberish is All You Need for Membership Inference Detection in Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2410.18371/) (2024)
- [The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models](https://mlsanity.com/paper/2408.01228/) (2024)
- [Defending Our Privacy With Backdoors](https://mlsanity.com/paper/2310.08320/) (2023)
- [CollaFuse: Collaborative Diffusion Models](https://mlsanity.com/paper/2406.14429/) (2024)
- [LeakyCLIP: Extracting Training Data from CLIP](https://mlsanity.com/paper/2508.00756/) (2025)
- [EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection](https://mlsanity.com/paper/2506.09827/) (2025)
- [SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning](https://mlsanity.com/paper/2603.26316/) (2026)
- [Better Membership Inference Privacy Measurement through Discrepancy](https://mlsanity.com/paper/2405.15140/) (2024)
- [Position: Life-Logging Video Streams Make the Privacy-Utility Trade-off Inevitable](https://mlsanity.com/paper/2605.10404/) (2026)
- [Adversarial Attacks in Multimodal Systems: A Practitioner's Survey](https://mlsanity.com/paper/2505.03084/) (2025)
- [Multimodal Privacy-Preserving Entity Resolution with Fully Homomorphic Encryption](https://mlsanity.com/paper/2601.18612/) (2026)
- [Membership Inference for Contrastive Pre-training Models with Text-only PII Queries](https://mlsanity.com/paper/2603.14222/) (2026)
- [From Measurement to Mitigation: Quantifying and Reducing Identity Leakage in Image Representation Encoders with Linear Subspace Removal](https://mlsanity.com/paper/2604.05296/) (2026)
- [No Safe Dose: How Training Data Drives Unsafe Image Generation](https://mlsanity.com/paper/2605.28137/) (2026)
- [TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction](https://mlsanity.com/paper/2607.04484/) (2026)
- [Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models](https://mlsanity.com/paper/2609.00411/) (2026)
- [VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale](https://mlsanity.com/paper/2609.32016/) (2026)

## Abstract (arXiv)

With the rise of deep learning in various applications, privacy concerns around the protection of training data have become a critical area of research. Whereas prior studies have focused on privacy risks in single-modal models, we introduce a novel method to assess privacy for multi-modal models, specifically vision-language models like CLIP. The proposed Identity Inference Attack (IDIA) reveals whether an individual was included in the training data by querying the model with images of the same person. Letting the model choose from a wide variety of possible text labels, the model reveals whether it recognizes the person and, therefore, was used for training. Our large-scale experiments on CLIP demonstrate that individuals used for training can be identified with very high accuracy. We confirm that the model has learned to associate names with depicted individuals, implying the existence of sensitive information that can be extracted by adversaries. Our results highlight the need for stronger privacy protection in large-scale models and suggest that IDIAs can be used to prove the unauthorized use of data for training and to enforce privacy laws.