# What do we learn from inverting CLIP models?

arXiv: https://arxiv.org/abs/2403.02580 · HTML: https://mlsanity.com/paper/2403.02580/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications](https://mlsanity.com/paper/2108.02818/) (2021)
- [What do Vision Transformers Learn? A Visual Exploration](https://mlsanity.com/paper/2212.06727/) (2022)
- [CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes](https://mlsanity.com/paper/2304.06061/) (2023)
- [Into the LAIONs Den: Investigating Hate in Multimodal Datasets](https://mlsanity.com/paper/2311.03449/) (2023)
- [Analyzing Bias in Diffusion-based Face Generation Models](https://mlsanity.com/paper/2305.06402/) (2023)
- [Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations](https://mlsanity.com/paper/2201.12961/) (2022)
- [Identifying and Mitigating Model Failures through Few-shot CLIP-aided Diffusion Generation](https://mlsanity.com/paper/2312.05464/) (2023)

## Built on by (most cited first, 13 of 13)

- [X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP](https://mlsanity.com/paper/2505.05528/) (2025)
- [If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions](https://mlsanity.com/paper/2403.16442/) (2024)
- [Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics](https://mlsanity.com/paper/2502.11725/) (2025)
- [Implicit Inversion turns CLIP into a Decoder](https://mlsanity.com/paper/2505.23161/) (2025)
- [LeakyCLIP: Extracting Training Data from CLIP](https://mlsanity.com/paper/2508.00756/) (2025)
- [Unaligning Everything: Or Aligning Any Text to Any Image in Multimodal Models](https://mlsanity.com/paper/2407.01157/) (2024)
- [Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning](https://mlsanity.com/paper/2510.26311/) (2025)
- [Does CLIP perceive art the same way we do?](https://mlsanity.com/paper/2505.05229/) (2025)
- [Image Reconstruction as a Tool for Feature Analysis](https://mlsanity.com/paper/2506.07803/) (2025)
- [On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness](https://mlsanity.com/paper/2508.09814/) (2025)
- [If you can describe it, they can see it: Cross-Modal Learning of Visual Concepts from Textual Descriptions](https://mlsanity.com/paper/2411.15611/) (2024)
- [Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis](https://mlsanity.com/paper/2602.19756/) (2026)
- [TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction](https://mlsanity.com/paper/2607.04484/) (2026)

## Abstract (arXiv)

We employ an inversion-based approach to examine CLIP models. Our examination reveals that inverting CLIP models results in the generation of images that exhibit semantic alignment with the specified target prompts. We leverage these inverted images to gain insights into various aspects of CLIP models, such as their ability to blend concepts and inclusion of gender biases. We notably observe instances of NSFW (Not Safe For Work) images during model inversion. This phenomenon occurs even for semantically innocuous prompts, like "a beautiful landscape," as well as for prompts involving the names of celebrities.