# Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection

arXiv: https://arxiv.org/abs/2409.02664 · HTML: https://mlsanity.com/paper/2409.02664/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain](https://mlsanity.com/paper/2103.01856/) (2021)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection](https://mlsanity.com/paper/2101.01456/) (2021)
- [UCF: Uncovering Common Features for Generalizable Deepfake Detection](https://mlsanity.com/paper/2304.13949/) (2023)
- [Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection](https://mlsanity.com/paper/2203.12208/) (2022)
- [DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models](https://mlsanity.com/paper/2210.06998/) (2022)
- [DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection](https://mlsanity.com/paper/2307.01426/) (2023)
- [Raising the Bar of AI-generated Image Detection with CLIP](https://mlsanity.com/paper/2312.00195/) (2023)
- [Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection](https://mlsanity.com/paper/2311.11278/) (2023)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Arc2Face: A Foundation Model for ID-Consistent Human Faces](https://mlsanity.com/paper/2403.11641/) (2024)
- [TALL: Thumbnail Layout for Deepfake Video Detection](https://mlsanity.com/paper/2307.07494/) (2023)
- [LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection](https://mlsanity.com/paper/2401.13856/) (2024)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes](https://mlsanity.com/paper/2211.11296/) (2022)
- [Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detection](https://mlsanity.com/paper/2403.01786/) (2024)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [TransFace++: Rethinking the Face Recognition Paradigm with a Focus on Accuracy, Efficiency, and Security](https://mlsanity.com/paper/2308.10133/) (2023)

## Built on by (most cited first, 30 of 30)

- [Dual Data Alignment Makes AI-Generated Image Detector Easier Generalizable](https://mlsanity.com/paper/2505.14359/) (2025)
- [X2-DFD: A framework for eXplainable and eXtendable Deepfake Detection](https://mlsanity.com/paper/2410.06126/) (2024)
- [Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection](https://mlsanity.com/paper/2503.14853/) (2025)
- [Breaking Latent Prior Bias in Detectors for Generalizable AIGC Image Detection](https://mlsanity.com/paper/2506.00874/) (2025)
- [Towards Generalizable AI-Generated Image Detection via Image-Adaptive Prompt Learning](https://mlsanity.com/paper/2508.01603/) (2025)
- [Rethinking Cross-Generator Image Forgery Detection through DINOv3](https://mlsanity.com/paper/2511.22471/) (2025)
- [Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes](https://mlsanity.com/paper/2505.19582/) (2025)
- [DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detection](https://mlsanity.com/paper/2511.13108/) (2025)
- [AlignGemini: Generalizable AI-Generated Image Detection Through Task-Model Alignment](https://mlsanity.com/paper/2512.06746/) (2025)
- [AgentFoX: LLM Agent-Guided Fusion with eXplainability for AI-Generated Image Detection](https://mlsanity.com/paper/2603.23115/) (2026)
- [Deep Residual Injection for Full-Spectrum Forensic Signal Perception in Multimodal Large Language Models](https://mlsanity.com/paper/2606.15880/) (2026)
- [Forensics Adapter: Unleashing CLIP for Generalizable Face Forgery Detection](https://mlsanity.com/paper/2411.19715/) (2024)
- [Fine-Grained DINO Tuning with Dual Supervision for Face Forgery Detection](https://mlsanity.com/paper/2511.12107/) (2025)
- [AIFIND: Artifact-Aware Interpreting Fine-Grained Alignment for Incremental Face Forgery Detection](https://mlsanity.com/paper/2604.16207/) (2026)
- [Detecting AI-Generated Video: A Vision-Language Dual-View Survey](https://mlsanity.com/paper/2607.10787/) (2026)
- [Uncertainty-Aware Deepfake Detection via Multi-View Structural Learning](https://mlsanity.com/paper/2607.28769/) (2026)
- [Revisiting Face Forgery Detection: From Facial Representation to Forgery Detection](https://mlsanity.com/paper/2409.16945/) (2024)
- [HAMLET-FFD: Hierarchical Adaptive Multi-modal Learning Embeddings Transformation for Face Forgery Detection](https://mlsanity.com/paper/2507.20913/) (2025)
- [Cognitive Inception: Agentic Reasoning against Visual Deceptions by Injecting Skepticism](https://mlsanity.com/paper/2511.17672/) (2025)
- [Low-rank Orthogonal Subspace Intervention for Generalizable Face Forgery Detection](https://mlsanity.com/paper/2601.11915/) (2026)
- [Unleashing Vision-Language Semantics for Deepfake Video Detection](https://mlsanity.com/paper/2603.24454/) (2026)
- [GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection](https://mlsanity.com/paper/2603.29295/) (2026)
- [LAA-X: Unified Localized Artifact Attention for Quality-Agnostic and Generalizable Face Forgery Detection](https://mlsanity.com/paper/2604.04086/) (2026)
- [M3D-Net: Multi-Modal 3D Facial Feature Reconstruction Network for Deepfake Detection](https://mlsanity.com/paper/2604.14574/) (2026)
- [DVAR: Adversarial Multi-Agent Debate for Video Authenticity Detection](https://mlsanity.com/paper/2604.16987/) (2026)
- [Generalizable Face Forgery Detection via Separable Prompt Learning](https://mlsanity.com/paper/2604.17307/) (2026)
- [FairForensics: Seeing Expressions and Parsing Demographics via Vision-Language Modeling for Generalizable Fair Deepfake Detection](https://mlsanity.com/paper/2608.01661/) (2026)
- [Environment-Invariant Subspace Learning for Generalizable Deepfake Detection](https://mlsanity.com/paper/2608.17700/) (2026)
- [Beyond Natural Images: Rethinking AI-Generated Image Detection in Documents](https://mlsanity.com/paper/2609.14352/) (2026)
- [Revisiting Cross-Reconstruction for Generalizable Deepfake Detection](https://mlsanity.com/paper/2610.01544/) (2026)

## Abstract (arXiv)

The proliferation of deepfake faces poses huge potential negative impacts on our daily lives. Despite substantial advancements in deepfake detection over these years, the generalizability of existing methods against forgeries from unseen datasets or created by emerging generative models remains constrained. In this paper, inspired by the zero-shot advantages of Vision-Language Models (VLMs), we propose a novel approach that repurposes a well-trained VLM for general deepfake detection. Motivated by the model reprogramming paradigm that manipulates the model prediction via input perturbations, our method can reprogram a pre-trained VLM model (e.g., CLIP) solely based on manipulating its input without tuning the inner parameters. First, learnable visual perturbations are used to refine feature extraction for deepfake detection. Then, we exploit information of face embedding to create sample-level adaptative text prompts, improving the performance. Extensive experiments on several popular benchmark datasets demonstrate that (1) the cross-dataset and cross-manipulation performances of deepfake detection can be significantly and consistently improved (e.g., over 88% AUC in cross-dataset setting from FF++ to WildDeepfake); (2) the superior performances are achieved with fewer trainable parameters, making it a promising approach for real-world applications.