# DeCLIP: Decoding CLIP representations for deepfake localization

arXiv: https://arxiv.org/abs/2409.08849 · HTML: https://mlsanity.com/paper/2409.08849/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 75 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [On the detection of synthetic images generated by diffusion models](https://mlsanity.com/paper/2211.00680/) (2022)
- [What makes fake images detectable? Understanding properties that generalize](https://mlsanity.com/paper/2008.10588/) (2020)
- [On the Detection of Digital Face Manipulation](https://mlsanity.com/paper/1910.01717/) (2019)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [Media Forensics and DeepFakes: an overview](https://mlsanity.com/paper/2001.06564/) (2020)
- [Do GANs leave artificial fingerprints?](https://mlsanity.com/paper/1812.11842/) (2018)
- [Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection](https://mlsanity.com/paper/2312.16649/) (2023)
- [Raising the Bar of AI-generated Image Detection with CLIP](https://mlsanity.com/paper/2312.00195/) (2023)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Fighting Fake News: Image Splice Detection via Learned Self-Consistency](https://mlsanity.com/paper/1805.04096/) (2018)
- [Hierarchical Fine-Grained Image Forgery Detection and Localization](https://mlsanity.com/paper/2303.17111/) (2023)
- [PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization](https://mlsanity.com/paper/2103.10596/) (2021)

## Built on by (most cited first, 17 of 17)

- [Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning](https://mlsanity.com/paper/2412.00175/) (2024)
- [FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning](https://mlsanity.com/paper/2412.12032/) (2024)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [Human Action CLIPs: Detecting AI-generated Human Motion](https://mlsanity.com/paper/2412.00526/) (2024)
- [Multimodal Medical Image Binding via Shared Text Embeddings](https://mlsanity.com/paper/2506.18072/) (2025)
- [Processing and acquisition traces in visual encoders: What does CLIP know about your camera?](https://mlsanity.com/paper/2508.10637/) (2025)
- [Who Made This? Fake Detection and Source Attribution with Diffusion Features](https://mlsanity.com/paper/2510.27602/) (2025)
- [uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data](https://mlsanity.com/paper/2511.13036/) (2025)
- [Investigating self-supervised representations for audio-visual deepfake detection](https://mlsanity.com/paper/2511.17181/) (2025)
- [TAP into the Patch Tokens: Leveraging Vision Foundation Model Features for AI-Generated Image Detection](https://mlsanity.com/paper/2604.26772/) (2026)
- [AmCLR: Unified Augmented Learning for Cross-Modal Representations](https://mlsanity.com/paper/2412.07979/) (2024)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [Beyond Flicker: Detecting Kinematic Inconsistencies for Generalizable Deepfake Video Detection](https://mlsanity.com/paper/2512.04175/) (2025)
- [Suppressing Forgery-Specific Shortcuts for Generalizable Deepfake Detection](https://mlsanity.com/paper/2606.01843/) (2026)
- [Chroma Clues: Leveraging Color Statistics to Detect Synthetic Images](https://mlsanity.com/paper/2606.02224/) (2026)
- [Fast Object Removal Attacks on Safety-Critical Video-based Perception Systems](https://mlsanity.com/paper/2608.02806/) (2026)
- [Invisible Shortcuts: Why Vision Encoders Know Your Camera](https://mlsanity.com/paper/2608.05424/) (2026)

## Abstract (arXiv)

Generative models can create entirely new images, but they can also partially modify real images in ways that are undetectable to the human eye. In this paper, we address the challenge of automatically detecting such local manipulations. One of the most pressing problems in deepfake detection remains the ability of models to generalize to different classes of generators. In the case of fully manipulated images, representations extracted from large self-supervised models (such as CLIP) provide a promising direction towards more robust detectors. Here, we introduce DeCLIP, a first attempt to leverage such large pretrained features for detecting local manipulations. We show that, when combined with a reasonably large convolutional decoder, pretrained self-supervised representations are able to perform localization and improve generalization capabilities over existing methods. Unlike previous work, our approach is able to perform localization on the challenging case of latent diffusion models, where the entire image is affected by the fingerprint of the generator. Moreover, we observe that this type of data, which combines local semantic information with a global fingerprint, provides more stable generalization than other categories of generative methods.