# AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors

arXiv: https://arxiv.org/abs/2310.17419 · HTML: https://mlsanity.com/paper/2310.17419/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 73 of its 99 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 73)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)

## Built on by (most cited first, 30 of 64)

- [VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models](https://mlsanity.com/paper/2403.06098/) (2024)
- [SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model](https://mlsanity.com/paper/2412.04292/) (2024)
- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation](https://mlsanity.com/paper/2503.14905/) (2025)
- [A Bias-Free Training Paradigm for More General AI-generated Image Detection](https://mlsanity.com/paper/2412.17671/) (2024)
- [FIRE: Robust Detection of Diffusion-Generated Images via Frequency-Guided Reconstruction Error](https://mlsanity.com/paper/2412.07140/) (2024)
- [FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Models](https://mlsanity.com/paper/2404.13306/) (2024)
- [AI-Generated Video Detection via Perceptual Straightening](https://mlsanity.com/paper/2507.00583/) (2025)
- [BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanation](https://mlsanity.com/paper/2505.12620/) (2025)
- [AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models](https://mlsanity.com/paper/2507.02664/) (2025)
- [BusterX++: Towards Unified Cross-Modal AI-Generated Content Detection and Explanation with MLLM](https://mlsanity.com/paper/2507.14632/) (2025)
- [Safety Alignment for Vision Language Models](https://mlsanity.com/paper/2405.13581/) (2024)
- [Bridging the Gap Between Ideal and Real-world Evaluation: Benchmarking AI-Generated Image Detection in Challenging Scenarios](https://mlsanity.com/paper/2509.09172/) (2025)
- [Bi-LORA: A Vision-Language Approach for Synthetic Image Detection](https://mlsanity.com/paper/2404.01959/) (2024)
- [Toward Generalizable Forgery Detection and Reasoning](https://mlsanity.com/paper/2503.21210/) (2025)
- [DeCLIP: Decoding CLIP representations for deepfake localization](https://mlsanity.com/paper/2409.08849/) (2024)
- [Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models](https://mlsanity.com/paper/2503.15024/) (2025)
- [Vulnerabilities in AI-generated Image Detection: The Challenge of Adversarial Attacks](https://mlsanity.com/paper/2407.20836/) (2024)
- [T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognition](https://mlsanity.com/paper/2409.19734/) (2024)
- [FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensics](https://mlsanity.com/paper/2503.24267/) (2025)
- [Navigating the Challenges of AI-Generated Image Detection in the Wild: What Truly Matters?](https://mlsanity.com/paper/2507.10236/) (2025)
- [Interpretable and Reliable Detection of AI-Generated Images via Grounded Reasoning in MLLMs](https://mlsanity.com/paper/2506.07045/) (2025)
- [RAIDX: A Retrieval-Augmented Generation and GRPO Reinforcement Learning Framework for Explainable Deepfake Detection](https://mlsanity.com/paper/2508.04524/) (2025)
- [Venus-DeFakerOne: Unified Fake Image Detection & Localization](https://mlsanity.com/paper/2605.14091/) (2026)
- [DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models](https://mlsanity.com/paper/2506.03007/) (2025)
- [ThinkFake: Reasoning in Multimodal Large Language Models for AI-Generated Image Detection](https://mlsanity.com/paper/2509.19841/) (2025)
- [Multiverse Through Deepfakes: The MultiFakeVerse Dataset of Person-Centric Visual and Conceptual Manipulations](https://mlsanity.com/paper/2506.00868/) (2025)
- [From Evidence to Verdict: An Agent-Based Forensic Framework for AI-Generated Image Detection](https://mlsanity.com/paper/2511.00181/) (2025)
- [REVEAL: Reasoning-Enhanced Forensic Evidence Analysis for Explainable AI-Generated Image Detection](https://mlsanity.com/paper/2511.23158/) (2025)
- [A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning](https://mlsanity.com/paper/2410.00485/) (2024)

## Abstract (arXiv)

Deep generative models can create remarkably photorealistic fake images while raising concerns about misinformation and copyright infringement, known as deepfake threats. Deepfake detection technique is developed to distinguish between real and fake images, where the existing methods typically learn classifiers in the image domain or various feature domains. However, the generalizability of deepfake detection against emerging and more advanced generative models remains challenging. In this paper, being inspired by the zero-shot advantages of Vision-Language Models (VLMs), we propose a novel approach called AntifakePrompt, using VLMs (e.g., InstructBLIP) and prompt tuning techniques to improve the deepfake detection accuracy over unseen data. We formulate deepfake detection as a visual question answering problem, and tune soft prompts for InstructBLIP to answer the real/fake information of a query image. We conduct full-spectrum experiments on datasets from a diversity of 3 held-in and 20 held-out generative models, covering modern text-to-image generation, image editing and adversarial image attacks. These testing datasets provide useful benchmarks in the realm of deepfake detection for further research. Moreover, results demonstrate that (1) the deepfake detection accuracy can be significantly and consistently improved (from 71.06% to 92.11%, in average accuracy over unseen domains) using pretrained vision-language models with prompt tuning; (2) our superior performance is at less cost of training data and trainable parameters, resulting in an effective and efficient solution for deepfake detection. Code and models can be found at https://github.com/nctu-eva-lab/AntifakePrompt.