Data to 7 Oct 2026

FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant

arXiv 2408.10072 ↗First posted 19 Aug 2024cs.CV cs.AIcited by 59

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

The rapid advancement of deepfake technologies has sparked widespread public concern, particularly as face forgery poses a serious threat to public information security. However, the unknown and diverse forgery techniques, varied facial features and complex environmental factors pose significant challenges for face forgery analysis. Existing datasets lack descriptive annotations of these aspects, making it difficult for models to distinguish between real and forged faces using only visual information amid various confounding factors. In addition, existing methods fail to yield user-friendly and explainable results, hindering the understanding of the model's decision-making process. To address these challenges, we introduce a novel Open-World Face Forgery Analysis VQA (OW-FFA-VQA) task and its corresponding benchmark. To tackle this task, we first establish a dataset featuring a diverse collection of real and forged face images with essential descriptions and reliable forgery reasoning. Based on this dataset, we introduce FFAA: Face Forgery Analysis Assistant, consisting of a fine-tuned Multimodal Large Language Model (MLLM) and Multi-answer Intelligent Decision System (MIDS). By integrating hypothetical prompts with MIDS, the impact of fuzzy classification boundaries is effectively mitigated, enhancing model robustness. Extensive experiments demonstrate that our method not only provides user-friendly and explainable results but also significantly boosts accuracy and robustness compared to previous methods.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 34 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

24 of 34 references.

Closest ancestors · 1

Same direction, newest first

2024CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection20 Feb

Other references · 23

2015Deep Residual Learning for Image Recognitioncited by 51,809
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
2023Improved Baselines with Visual Instruction Tuningcited by 4,362
2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,095
2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000

11 more

2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
2019Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensicscited by 573
2021Multi-attentional Deepfake Detectioncited by 339
2019On the Detection of Digital Face Manipulationcited by 259
2020DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detectioncited by 251
2020DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detectioncited by 231
2022Implicit Identity Leakage: The Stumbling Block to Improving Deepfake Detection Generalizationcited by 122
2024Common Sense Reasoning for Deepfake Detectioncited by 64
2023GANDiffFace: Controllable Generation of Synthetic Datasets for Face Recognition with Realistic Variationscited by 49
2024Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detectioncited by 47
2022FedForgery: Generalized Face Forgery Detection with Residual Federated Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 33; 2026: 18.

59 papers.

Sort:

By area:

2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
2024FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Modelscited by 111
2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
2025Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanationcited by 66
2025LEGION: Learning to Ground and Explain for Synthetic Image Detectioncited by 65
2024X2-DFD: A framework for eXplainable and eXtendable Deepfake Detectioncited by 35
2025AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Modelscited by 33
2025Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoningcited by 23
2025Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detectioncited by 20
2025Toward Generalizable Forgery Detection and Reasoningcited by 20
2025Survey on AI-Generated Media Detection: From Non-MLLM to MLLMcited by 12
2025VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Modelscited by 12

38 more

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way