# PUDD: Towards Robust Multi-modal Prototype-based Deepfake Detection

arXiv: https://arxiv.org/abs/2406.15921 · HTML: https://mlsanity.com/paper/2406.15921/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 22 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Images](https://mlsanity.com/paper/2303.14126/) (2023)
- [Evading DeepFake Detectors via Adversarial Statistical Consistency](https://mlsanity.com/paper/2304.11670/) (2023)
- [Detecting Deepfakes Without Seeing Any](https://mlsanity.com/paper/2311.01458/) (2023)
- [Domain Adaptation and Autoencoder Based Unsupervised Speech Enhancement](https://mlsanity.com/paper/2112.05036/) (2021)
- [Self-Supervised Graph Transformer for Deepfake Detection](https://mlsanity.com/paper/2307.15019/) (2023)
- [ProtoExplorer: Interpretable Forensic Analysis of Deepfake Videos using Prototype Exploration and Refinement](https://mlsanity.com/paper/2309.11155/) (2023)
- [Artifact magnification on deepfake videos increases human detection and subjective confidence](https://mlsanity.com/paper/2304.04733/) (2023)

## Built on by (most cited first, 10 of 10)

- [BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanation](https://mlsanity.com/paper/2505.12620/) (2025)
- [Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content](https://mlsanity.com/paper/2412.12278/) (2024)
- [Few-Shot Learner Generalizes Across AI-Generated Image Detection](https://mlsanity.com/paper/2501.08763/) (2025)
- [Robust Single-Stage Fully Sparse 3D Object Detection via Detachable Latent Diffusion](https://mlsanity.com/paper/2508.03252/) (2025)
- [Prototype-Based Continual Learning with Label-free Replay Buffer and Cluster Preservation Loss](https://mlsanity.com/paper/2504.07240/) (2025)
- [Self-Supervised Representation Learning for Adversarial Attack Detection](https://mlsanity.com/paper/2407.04382/) (2024)
- [Seeing Through Deepfakes: A Human-Inspired Framework for Multi-Face Detection](https://mlsanity.com/paper/2507.14807/) (2025)
- [Fine-Grained DINO Tuning with Dual Supervision for Face Forgery Detection](https://mlsanity.com/paper/2511.12107/) (2025)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [ForensicFlow: A Tri-Modal Adaptive Network for Robust Deepfake Detection](https://mlsanity.com/paper/2511.14554/) (2025)

## Abstract (arXiv)

Deepfake techniques generate highly realistic data, making it challenging for humans to discern between actual and artificially generated images. Recent advancements in deep learning-based deepfake detection methods, particularly with diffusion models, have shown remarkable progress. However, there is a growing demand for real-world applications to detect unseen individuals, deepfake techniques, and scenarios. To address this limitation, we propose a Prototype-based Unified Framework for Deepfake Detection (PUDD). PUDD offers a detection system based on similarity, comparing input data against known prototypes for video classification and identifying potential deepfakes or previously unseen classes by analyzing drops in similarity. Our extensive experiments reveal three key findings: (1) PUDD achieves an accuracy of 95.1% on Celeb-DF, outperforming state-of-the-art deepfake detection methods; (2) PUDD leverages image classification as the upstream task during training, demonstrating promising performance in both image classification and deepfake detection tasks during inference; (3) PUDD requires only 2.7 seconds for retraining on new data and emits 105 times less carbon compared to the state-of-the-art model, making it significantly more environmentally friendly.