# Combining EfficientNet and Vision Transformers for Video Deepfake Detection

arXiv: https://arxiv.org/abs/2107.02612 · HTML: https://mlsanity.com/paper/2107.02612/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 29 of 29)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection](https://mlsanity.com/paper/2001.03024/) (2020)
- [DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection](https://mlsanity.com/paper/2001.00179/) (2020)
- [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://mlsanity.com/paper/1812.08685/) (2018)
- [TweepFake: about Detecting Deepfake Tweets](https://mlsanity.com/paper/2008.00036/) (2020)
- [DeepFake Detection by Analyzing Convolutional Traces](https://mlsanity.com/paper/2004.10448/) (2020)
- [Deepfake Video Detection Using Convolutional Vision Transformer](https://mlsanity.com/paper/2102.11126/) (2021)
- [Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoders](https://mlsanity.com/paper/2008.05231/) (2020)
- [Efficient Document Re-Ranking for Transformers by Precomputing Term Representations](https://mlsanity.com/paper/2004.14255/) (2020)
- [Fighting deepfakes by detecting GAN DCT anomalies](https://mlsanity.com/paper/2101.09781/) (2021)
- [Deepfakes Detection with Automatic Face Weighting](https://mlsanity.com/paper/2004.12027/) (2020)
- [Deepfake Detection using Spatiotemporal Convolutional Networks](https://mlsanity.com/paper/2006.14749/) (2020)
- [Deepfake Detection Scheme Based on Vision Transformer and Distillation](https://mlsanity.com/paper/2104.01353/) (2021)
- [Transformer Reasoning Network for Image-Text Matching and Retrieval](https://mlsanity.com/paper/2004.09144/) (2020)
- [Virtual to Real adaptation of Pedestrian Detectors](https://mlsanity.com/paper/2001.03032/) (2020)
- [Towards Efficient Cross-Modal Visual Textual Retrieval using Transformer-Encoder Deep Features](https://mlsanity.com/paper/2106.00358/) (2021)

## Built on by (most cited first, 30 of 82)

- [TALL: Thumbnail Layout for Deepfake Video Detection](https://mlsanity.com/paper/2307.07494/) (2023)
- [LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection](https://mlsanity.com/paper/2401.13856/) (2024)
- [AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset](https://mlsanity.com/paper/2311.15308/) (2023)
- [Common Sense Reasoning for Deepfake Detection](https://mlsanity.com/paper/2402.00126/) (2024)
- [Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization](https://mlsanity.com/paper/2305.01979/) (2023)
- [Improving Fairness in Deepfake Detection](https://mlsanity.com/paper/2306.16635/) (2023)
- [UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2308.14395/) (2023)
- [MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection](https://mlsanity.com/paper/2211.10996/) (2022)
- [Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning](https://mlsanity.com/paper/2412.00175/) (2024)
- [Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2203.01536/) (2022)
- [Learning Spatiotemporal Inconsistency via Thumbnail Layout for Face Deepfake Detection](https://mlsanity.com/paper/2403.10261/) (2024)
- [Deepfake Video Detection with Spatiotemporal Dropout Transformer](https://mlsanity.com/paper/2207.06612/) (2022)
- [Deep Convolutional Pooling Transformer for Deepfake Detection](https://mlsanity.com/paper/2209.05299/) (2022)
- [Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders](https://mlsanity.com/paper/2310.20704/) (2023)
- [PolyGlotFake: A Novel Multilingual and Multimodal DeepFake Dataset](https://mlsanity.com/paper/2405.08838/) (2024)
- [Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis](https://mlsanity.com/paper/2405.00355/) (2024)
- [Online Segmentation of LiDAR Sequences: Dataset and Algorithm](https://mlsanity.com/paper/2206.08194/) (2022)
- [A Comprehensive Survey of Transformers for Computer Vision](https://mlsanity.com/paper/2211.06004/) (2022)
- [SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework](https://mlsanity.com/paper/2401.04364/) (2024)
- [A Review of Deep Learning-based Approaches for Deepfake Content Detection](https://mlsanity.com/paper/2202.06095/) (2022)
- [GenConViT: Deepfake Video Detection Using Generative Convolutional Vision Transformer](https://mlsanity.com/paper/2307.07036/) (2023)
- [Detecting Images Generated by Diffusers](https://mlsanity.com/paper/2303.05275/) (2023)
- [GenFace: A Large-Scale Fine-Grained Face Forgery Benchmark and Cross Appearance-Edge Learning](https://mlsanity.com/paper/2402.02003/) (2024)
- [DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization](https://mlsanity.com/paper/2411.10193/) (2024)
- [MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection](https://mlsanity.com/paper/2409.09724/) (2024)
- [GenAI Mirage: The Impostor Bias and the Deepfake Detection Challenge in the Era of Artificial Illusions](https://mlsanity.com/paper/2312.16220/) (2023)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [Recurrent Vision Transformer for Solving Visual Reasoning Problems](https://mlsanity.com/paper/2111.14576/) (2021)
- [VideoFACT: Detecting Video Forgeries Using Attention, Scene Context, and Forensic Traces](https://mlsanity.com/paper/2211.15775/) (2022)
- [MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark](https://mlsanity.com/paper/2505.11109/) (2025)

## Abstract (arXiv)

Deepfakes are the result of digital manipulation to forge realistic yet fake imagery. With the astonishing advances in deep generative models, fake images or videos are nowadays obtained using variational autoencoders (VAEs) or Generative Adversarial Networks (GANs). These technologies are becoming more accessible and accurate, resulting in fake videos that are very difficult to be detected. Traditionally, Convolutional Neural Networks (CNNs) have been used to perform video deepfake detection, with the best results obtained using methods based on EfficientNet B7. In this study, we focus on video deep fake detection on faces, given that most methods are becoming extremely accurate in the generation of realistic human faces. Specifically, we combine various types of Vision Transformers with a convolutional EfficientNet B0 used as a feature extractor, obtaining comparable results with some very recent methods that use Vision Transformers. Differently from the state-of-the-art approaches, we use neither distillation nor ensemble methods. Furthermore, we present a straightforward inference procedure based on a simple voting scheme for handling multiple faces in the same video shot. The best model achieved an AUC of 0.951 and an F1 score of 88.0%, very close to the state-of-the-art on the DeepFake Detection Challenge (DFDC).