# Hybrid Transformer Network for Deepfake Detection

arXiv: https://arxiv.org/abs/2208.05820 · HTML: https://mlsanity.com/paper/2208.05820/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals](https://mlsanity.com/paper/1901.02212/) (2019)
- [Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos](https://mlsanity.com/paper/1906.06876/) (2019)
- [ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis](https://mlsanity.com/paper/2103.05630/) (2021)
- [Recurrent Convolutional Strategies for Face Manipulation Detection in Videos](https://mlsanity.com/paper/1905.00582/) (2019)
- [Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues](https://mlsanity.com/paper/2003.06711/) (2020)
- [Deepfake Video Detection Using Convolutional Vision Transformer](https://mlsanity.com/paper/2102.11126/) (2021)
- [DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms](https://mlsanity.com/paper/2006.07634/) (2020)
- [Face Forgery Detection by 3D Decomposition](https://mlsanity.com/paper/2011.09737/) (2020)
- [On the generalization of GAN image forensics](https://mlsanity.com/paper/1902.11153/) (2019)
- [Video Transformer for Deepfake Detection with Incremental Learning](https://mlsanity.com/paper/2108.05307/) (2021)
- [Training Strategies and Data Augmentations in CNN-based DeepFake Video Detection](https://mlsanity.com/paper/2011.07792/) (2020)

## Built on by (most cited first, 6 of 6)

- [MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection](https://mlsanity.com/paper/2211.10996/) (2022)
- [Deep Convolutional Pooling Transformer for Deepfake Detection](https://mlsanity.com/paper/2209.05299/) (2022)
- [GenConViT: Deepfake Video Detection Using Generative Convolutional Vision Transformer](https://mlsanity.com/paper/2307.07036/) (2023)
- [AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection](https://mlsanity.com/paper/2310.13103/) (2023)
- [Detecting Localized Deepfake Manipulations Using Action Unit-Guided Video Representations](https://mlsanity.com/paper/2503.22121/) (2025)
- [AWARE-NET: Adaptive Weighted Averaging for Robust Ensemble Network in Deepfake Detection](https://mlsanity.com/paper/2505.00312/) (2025)

## Abstract (arXiv)

Deepfake media is becoming widespread nowadays because of the easily available tools and mobile apps which can generate realistic looking deepfake videos/images without requiring any technical knowledge. With further advances in this field of technology in the near future, the quantity and quality of deepfake media is also expected to flourish, while making deepfake media a likely new practical tool to spread mis/disinformation. Because of these concerns, the deepfake media detection tools are becoming a necessity. In this study, we propose a novel hybrid transformer network utilizing early feature fusion strategy for deepfake video detection. Our model employs two different CNN networks, i.e., (1) XceptionNet and (2) EfficientNet-B4 as feature extractors. We train both feature extractors along with the transformer in an end-to-end manner on FaceForensics++, DFDC benchmarks. Our model, while having relatively straightforward architecture, achieves comparable results to other more advanced state-of-the-art approaches when evaluated on FaceForensics++ and DFDC benchmarks. Besides this, we also propose novel face cut-out augmentations, as well as random cut-out augmentations. We show that the proposed augmentations improve the detection performance of our model and reduce overfitting. In addition to that, we show that our model is capable of learning from considerably small amount of data.