# Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection

arXiv: https://arxiv.org/abs/2503.14853 · HTML: https://mlsanity.com/paper/2503.14853/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 41)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [Detecting and Preventing Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2308.06394/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain](https://mlsanity.com/paper/2103.01856/) (2021)
- [UCF: Uncovering Common Features for Generalizable Deepfake Detection](https://mlsanity.com/paper/2304.13949/) (2023)
- [Learning Self-Consistency for Deepfake Detection](https://mlsanity.com/paper/2012.09311/) (2020)
- [AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models](https://mlsanity.com/paper/2308.15366/) (2023)
- [DF40: Toward Next-Generation Deepfake Detection](https://mlsanity.com/paper/2406.13495/) (2024)
- [Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection](https://mlsanity.com/paper/2311.11278/) (2023)
- [Implicit Identity Leakage: The Stumbling Block to Improving Deepfake Detection Generalization](https://mlsanity.com/paper/2210.14457/) (2022)
- [TALL: Thumbnail Layout for Deepfake Video Detection](https://mlsanity.com/paper/2307.07494/) (2023)
- [LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection](https://mlsanity.com/paper/2401.13856/) (2024)
- [FrePGAN: Robust Deepfake Detection Using Frequency-level Perturbations](https://mlsanity.com/paper/2202.03347/) (2022)

## Built on by (most cited first, 21 of 21)

- [AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models](https://mlsanity.com/paper/2507.02664/) (2025)
- [EvoGuard: An Extensible Agentic RL-based Framework for Practical and Evolving AI-Generated Image Detection](https://mlsanity.com/paper/2603.17343/) (2026)
- [Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities](https://mlsanity.com/paper/2604.07763/) (2026)
- [TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection](https://mlsanity.com/paper/2512.10652/) (2025)
- [Explainable Deepfake Detection with RL Enhanced Self-Blended Images](https://mlsanity.com/paper/2601.15624/) (2026)
- [AgentFoX: LLM Agent-Guided Fusion with eXplainability for AI-Generated Image Detection](https://mlsanity.com/paper/2603.23115/) (2026)
- [VRAG-DFD: Verifiable Retrieval-Augmentation for MLLM-based Deepfake Detection](https://mlsanity.com/paper/2604.13660/) (2026)
- [Fine-Grained DINO Tuning with Dual Supervision for Face Forgery Detection](https://mlsanity.com/paper/2511.12107/) (2025)
- [RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetry](https://mlsanity.com/paper/2603.01544/) (2026)
- [Detecting AI-Generated Video: A Vision-Language Dual-View Survey](https://mlsanity.com/paper/2607.10787/) (2026)
- [Training-Free Multimodal Deepfake Detection via Graph Reasoning](https://mlsanity.com/paper/2509.21774/) (2025)
- [Explainable AI-Generated Image Detection RewardBench](https://mlsanity.com/paper/2511.12363/) (2025)
- [Cognitive Inception: Agentic Reasoning against Visual Deceptions by Injecting Skepticism](https://mlsanity.com/paper/2511.17672/) (2025)
- [Na"ive Exposure of Generative AI Capabilities Undermines Deepfake Detection](https://mlsanity.com/paper/2603.10504/) (2026)
- [High-Fidelity Face Content Recovery via Tamper-Resilient Versatile Watermarking](https://mlsanity.com/paper/2603.23940/) (2026)
- [Generalizable Face Forgery Detection via Separable Prompt Learning](https://mlsanity.com/paper/2604.17307/) (2026)
- [ID-Eraser: Proactive Defense Against Face Swapping via Identity Perturbation](https://mlsanity.com/paper/2604.21465/) (2026)
- [UniShield: Unified Face Attack Detection via KG-Informed Multimodal Reasoning](https://mlsanity.com/paper/2605.08709/) (2026)
- [Specialist-Generalist Fusion with Outcome-Supervised Rationales for Deepfake Detection](https://mlsanity.com/paper/2605.31192/) (2026)
- [ForensicZoom: Adaptive Visual Inspection with Multimodal LLMs for Industrial-Grade Face Forgery Detection](https://mlsanity.com/paper/2609.31661/) (2026)
- [EvoKnow: Continual Knowledge Evolution for AI-Generated Image Detection](https://mlsanity.com/paper/2610.11381/) (2026)

## Abstract (arXiv)

Current Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in understanding multimodal data, but their potential remains underexplored for deepfake detection due to the misalignment of their knowledge and forensics patterns. To this end, we present a novel framework that unlocks LVLMs' potential capabilities for deepfake detection. Our framework includes a Knowledge-guided Forgery Detector (KFD), a Forgery Prompt Learner (FPL), and a Large Language Model (LLM). The KFD is used to calculate correlations between image features and pristine/deepfake image description embeddings, enabling forgery classification and localization. The outputs of the KFD are subsequently processed by the Forgery Prompt Learner to construct fine-grained forgery prompt embeddings. These embeddings, along with visual and question prompt embeddings, are fed into the LLM to generate textual detection responses. Extensive experiments on multiple benchmarks, including FF++, CDF2, DFD, DFDCP, DFDC, and DF40, demonstrate that our scheme surpasses state-of-the-art methods in generalization performance, while also supporting multi-turn dialogue capabilities.