# Unlocking the Hidden Potential of CLIP in Generalizable Deepfake Detection

arXiv: https://arxiv.org/abs/2503.19683 · HTML: https://mlsanity.com/paper/2503.19683/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection](https://mlsanity.com/paper/2312.16649/) (2023)
- [Learning Self-Consistency for Deepfake Detection](https://mlsanity.com/paper/2012.09311/) (2020)
- [DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection](https://mlsanity.com/paper/2307.01426/) (2023)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection](https://mlsanity.com/paper/2311.11278/) (2023)
- [LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection](https://mlsanity.com/paper/2401.13856/) (2024)
- [Face Forensics in the Wild](https://mlsanity.com/paper/2103.16076/) (2021)
- [AltFreezing for More General Video Face Forgery Detection](https://mlsanity.com/paper/2307.08317/) (2023)
- [Exploiting Style Latent Flows for Generalizing Deepfake Video Detection](https://mlsanity.com/paper/2403.06592/) (2024)
- [CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection](https://mlsanity.com/paper/2402.12927/) (2024)
- [Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes](https://mlsanity.com/paper/2401.15668/) (2024)
- [Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing](https://mlsanity.com/paper/2501.04376/) (2025)
- [Learning Natural Consistency Representation for Face Forgery Video Detection](https://mlsanity.com/paper/2407.10550/) (2024)
- [Learning Spatiotemporal Inconsistency via Thumbnail Layout for Face Deepfake Detection](https://mlsanity.com/paper/2403.10261/) (2024)
- [Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Models](https://mlsanity.com/paper/2211.08682/) (2022)
- [Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis](https://mlsanity.com/paper/2405.00355/) (2024)

## Built on by (most cited first, 10 of 10)

- [PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysis](https://mlsanity.com/paper/2510.14241/) (2025)
- [AIFIND: Artifact-Aware Interpreting Fine-Grained Alignment for Incremental Face Forgery Detection](https://mlsanity.com/paper/2604.16207/) (2026)
- [Why Fake ? Unveiling the Semantic Vocabulary of Deepfake Detectors](https://mlsanity.com/paper/2607.07216/) (2026)
- [Leveraging Unlabeled Data from Unknown Sources via Dual-Path Guidance for Deepfake Face Detection](https://mlsanity.com/paper/2508.09022/) (2025)
- [Proto-LeakNet: Towards Signal-Leak Aware Attribution in Synthetic Human Face Imagery](https://mlsanity.com/paper/2511.04260/) (2025)
- [Koo-Fu CLIP: Closed-Form Adaptation of Vision-Language Models via Fukunaga-Koontz Linear Discriminant Analysis](https://mlsanity.com/paper/2602.01127/) (2026)
- [Segmentation-Guided Spatial Indexing for Generalizable and Explainable Deepfake Detection](https://mlsanity.com/paper/2606.00098/) (2026)
- [Suppressing Forgery-Specific Shortcuts for Generalizable Deepfake Detection](https://mlsanity.com/paper/2606.01843/) (2026)
- [The DeepSpeak-Agentic Dataset](https://mlsanity.com/paper/2606.03686/) (2026)
- [μFlow: Leveraging Average Images for Improving Generalisation of Deepfake Faces Detectors](https://mlsanity.com/paper/2606.30528/) (2026)

## Abstract (arXiv)

This paper tackles the challenge of detecting partially manipulated facial deepfakes, which involve subtle alterations to specific facial features while retaining the overall context, posing a greater detection difficulty than fully synthetic faces. We leverage the Contrastive Language-Image Pre-training (CLIP) model, specifically its ViT-L/14 visual encoder, to develop a generalizable detection method that performs robustly across diverse datasets and unknown forgery techniques with minimal modifications to the original model. The proposed approach utilizes parameter-efficient fine-tuning (PEFT) techniques, such as LN-tuning, to adjust a small subset of the model's parameters, preserving CLIP's pre-trained knowledge and reducing overfitting. A tailored preprocessing pipeline optimizes the method for facial images, while regularization strategies, including L2 normalization and metric learning on a hyperspherical manifold, enhance generalization. Trained on the FaceForensics++ dataset and evaluated in a cross-dataset fashion on Celeb-DF-v2, DFDC, FFIW, and others, the proposed method achieves competitive detection accuracy comparable to or outperforming much more complex state-of-the-art techniques. This work highlights the efficacy of CLIP's visual encoder in facial deepfake detection and establishes a simple, powerful baseline for future research, advancing the field of generalizable deepfake detection. The code is available at: https://github.com/yermandy/deepfake-detection