# CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection

arXiv: https://arxiv.org/abs/2402.12927 · HTML: https://mlsanity.com/paper/2402.12927/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [DIRE for Diffusion-Generated Image Detection](https://mlsanity.com/paper/2303.09295/) (2023)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)

## Built on by (most cited first, 30 of 47)

- [FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant](https://mlsanity.com/paper/2408.10072/) (2024)
- [Bridging the Gap Between Ideal and Real-world Evaluation: Benchmarking AI-Generated Image Detection in Challenging Scenarios](https://mlsanity.com/paper/2509.09172/) (2025)
- [FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning](https://mlsanity.com/paper/2412.12032/) (2024)
- [Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection](https://mlsanity.com/paper/2503.14853/) (2025)
- [Deepfake Detection that Generalizes Across Benchmarks](https://mlsanity.com/paper/2508.06248/) (2025)
- [Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Images](https://mlsanity.com/paper/2404.12908/) (2024)
- [DeCLIP: Decoding CLIP representations for deepfake localization](https://mlsanity.com/paper/2409.08849/) (2024)
- [GM-DF: Generalized Multi-Scenario Deepfake Detection](https://mlsanity.com/paper/2406.20078/) (2024)
- [ForgeLens: Data-Efficient Forgery Focus for Generalizable Forgery Image Detection](https://mlsanity.com/paper/2408.13697/) (2024)
- [VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models](https://mlsanity.com/paper/2503.06142/) (2025)
- [Unlocking the Hidden Potential of CLIP in Generalizable Deepfake Detection](https://mlsanity.com/paper/2503.19683/) (2025)
- [OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection](https://mlsanity.com/paper/2509.09495/) (2025)
- [FairAdapter: Detecting AI-generated Images with Improved Fairness](https://mlsanity.com/paper/2411.14755/) (2024)
- [CausalCLIP: Causally-Informed Feature Disentanglement and Filtering for Generalizable Detection of Generated Images](https://mlsanity.com/paper/2512.13285/) (2025)
- [Multimodal Conditional Information Bottleneck for Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2505.15217/) (2025)
- [Agent4FaceForgery: Multi-Agent LLM Framework for Realistic Face Forgery Detection](https://mlsanity.com/paper/2509.12546/) (2025)
- [Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach](https://mlsanity.com/paper/2408.16305/) (2024)
- [A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning](https://mlsanity.com/paper/2410.00485/) (2024)
- [Human Action CLIPs: Detecting AI-generated Human Motion](https://mlsanity.com/paper/2412.00526/) (2024)
- [Fully AI-Generated Image Detection: Definition, Recent Advances and Challenges](https://mlsanity.com/paper/2502.19716/) (2025)
- [Beyond Semantic Features: Pixel-level Mapping for Generalized AI-Generated Image Detection](https://mlsanity.com/paper/2512.17350/) (2025)
- [MGFFD-VLM: Multi-Granularity Prompt Learning for Face Forgery Detection with VLM](https://mlsanity.com/paper/2507.12232/) (2025)
- [Forensics Adapter: Unleashing CLIP for Generalizable Face Forgery Detection](https://mlsanity.com/paper/2411.19715/) (2024)
- [CorrDetail: Visual Detail Enhanced Self-Correction for Face Forgery Detection](https://mlsanity.com/paper/2507.05302/) (2025)
- [MiraGe: Multimodal Discriminative Representation Learning for Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2508.01525/) (2025)
- [Zero-Shot Visual Deepfake Detection: Can AI Predict and Prevent Fake Content Before It's Created?](https://mlsanity.com/paper/2509.18461/) (2025)
- [Investigating self-supervised representations for audio-visual deepfake detection](https://mlsanity.com/paper/2511.17181/) (2025)
- [Detecting AI-Generated Images via Distributional Deviations from Real Images](https://mlsanity.com/paper/2601.03586/) (2026)
- [Synthetic Image Detection with CLIP: Understanding and Assessing Predictive Cues](https://mlsanity.com/paper/2602.12381/) (2026)
- [Uncertainty-Aware Deepfake Detection via Multi-View Structural Learning](https://mlsanity.com/paper/2607.28769/) (2026)

## Abstract (arXiv)

The recent advancements in Generative Adversarial Networks (GANs) and the emergence of Diffusion models have significantly streamlined the production of highly realistic and widely accessible synthetic content. As a result, there is a pressing need for effective general purpose detection mechanisms to mitigate the potential risks posed by deepfakes. In this paper, we explore the effectiveness of pre-trained vision-language models (VLMs) when paired with recent adaptation methods for universal deepfake detection. Following previous studies in this domain, we employ only a single dataset (ProGAN) in order to adapt CLIP for deepfake detection. However, in contrast to prior research, which rely solely on the visual part of CLIP while ignoring its textual component, our analysis reveals that retaining the text part is crucial. Consequently, the simple and lightweight Prompt Tuning based adaptation strategy that we employ outperforms the previous SOTA approach by 5.01% mAP and 6.61% accuracy while utilizing less than one third of the training data (200k images as compared to 720k). To assess the real-world applicability of our proposed models, we conduct a comprehensive evaluation across various scenarios. This involves rigorous testing on images sourced from 21 distinct datasets, including those generated by GANs-based, Diffusion-based and Commercial tools.