# Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector

arXiv: https://arxiv.org/abs/2503.20188 · HTML: https://mlsanity.com/paper/2503.20188/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 86 of its 105 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 86)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)

## Built on by (most cited first, 30 of 40)

- [AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models](https://mlsanity.com/paper/2507.02664/) (2025)
- [Toward Generalizable Forgery Detection and Reasoning](https://mlsanity.com/paper/2503.21210/) (2025)
- [BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models](https://mlsanity.com/paper/2505.18132/) (2025)
- [50 Years of Automated Face Recognition](https://mlsanity.com/paper/2505.24247/) (2025)
- [When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning](https://mlsanity.com/paper/2602.08236/) (2026)
- [FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensics](https://mlsanity.com/paper/2503.24267/) (2025)
- [A Quality-Guided Mixture of Score-Fusion Experts Framework for Human Recognition](https://mlsanity.com/paper/2508.00053/) (2025)
- [Semantic Visual Anomaly Detection and Reasoning in AI-Generated Images](https://mlsanity.com/paper/2510.10231/) (2025)
- [Venus-DeFakerOne: Unified Fake Image Detection & Localization](https://mlsanity.com/paper/2605.14091/) (2026)
- [AvatarShield: Visual Reinforcement Learning for Human-Centric Synthetic Video Detection](https://mlsanity.com/paper/2505.15173/) (2025)
- [On the Holistic Approach for Detecting Human Image Forgery](https://mlsanity.com/paper/2601.04715/) (2026)
- [TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection](https://mlsanity.com/paper/2512.10652/) (2025)
- [MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models](https://mlsanity.com/paper/2601.20433/) (2026)
- [FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition](https://mlsanity.com/paper/2603.26908/) (2026)
- [Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs](https://mlsanity.com/paper/2509.25139/) (2025)
- [Explainable Deepfake Detection with RL Enhanced Self-Blended Images](https://mlsanity.com/paper/2601.15624/) (2026)
- [VIGIL: Part-Grounded Structured Reasoning for Generalizable Deepfake Detection](https://mlsanity.com/paper/2603.21526/) (2026)
- [Propose and Rectify: A Forensics-Driven MLLM Framework for Image Manipulation Localization](https://mlsanity.com/paper/2508.17976/) (2025)
- [Fake-in-Facext: Towards Fine-Grained Explainable DeepFake Analysis](https://mlsanity.com/paper/2510.20531/) (2025)
- [VRAG-DFD: Verifiable Retrieval-Augmentation for MLLM-based Deepfake Detection](https://mlsanity.com/paper/2604.13660/) (2026)
- [Explainable Deepfake Detection Challenge](https://mlsanity.com/paper/2607.21007/) (2026)
- [AuthGuard: Generalizable Deepfake Detection via Language Guidance](https://mlsanity.com/paper/2506.04501/) (2025)
- [EditSleuth: A Dataset of Grounded Reasoning Chains for Image-Edit Forensics](https://mlsanity.com/paper/2605.08695/) (2026)
- [HydraPrompt: An Adaptive and Asymmetric Framework of Vision-Language Models for Synthetic Image Detection](https://mlsanity.com/paper/2605.26421/) (2026)
- [Explainable Deepfake Detection with Feature-robust Augmentation and Evidence-grounded Explanation Optimization](https://mlsanity.com/paper/2608.20913/) (2026)
- [Leveraging Unlabeled Data from Unknown Sources via Dual-Path Guidance for Deepfake Face Detection](https://mlsanity.com/paper/2508.09022/) (2025)
- [Cognitive Inception: Agentic Reasoning against Visual Deceptions by Injecting Skepticism](https://mlsanity.com/paper/2511.17672/) (2025)
- [Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning](https://mlsanity.com/paper/2511.18104/) (2025)
- [Spoofing-aware Prompt Learning for Unified Physical-Digital Facial Attack Detection](https://mlsanity.com/paper/2512.06363/) (2025)
- [Na"ive Exposure of Generative AI Capabilities Undermines Deepfake Detection](https://mlsanity.com/paper/2603.10504/) (2026)

## Abstract (arXiv)

Deepfake detection is a long-established research topic vital for mitigating the spread of malicious misinformation. Unlike prior methods that provide either binary classification results or textual explanations separately, we introduce a novel method capable of generating both simultaneously. Our method harnesses the multi-modal learning capability of the pre-trained CLIP and the unprecedented interpretability of large language models (LLMs) to enhance both the generalization and explainability of deepfake detection. Specifically, we introduce a multi-modal face forgery detector (M2F2-Det) that employs tailored face forgery prompt learning, incorporating the pre-trained CLIP to improve generalization to unseen forgeries. Also, M2F2-Det incorporates an LLM to provide detailed textual explanations of its detection decisions, enhancing interpretability by bridging the gap between natural language and subtle cues of facial forgeries. Empirically, we evaluate M2F2-Det on both detection and explanation generation tasks, where it achieves state-of-the-art performance, demonstrating its effectiveness in identifying and explaining diverse forgeries.