# Deepfake Video Detection Using Convolutional Vision Transformer

arXiv: https://arxiv.org/abs/2102.11126 · HTML: https://mlsanity.com/paper/2102.11126/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 64 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection](https://mlsanity.com/paper/2001.00179/) (2020)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://mlsanity.com/paper/1812.08685/) (2018)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals](https://mlsanity.com/paper/1901.02212/) (2019)
- [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019)
- [Towards Automatic Face-to-Face Translation](https://mlsanity.com/paper/2003.00418/) (2020)
- [BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs](https://mlsanity.com/paper/1907.05047/) (2019)
- [World-Consistent Video-to-Video Synthesis](https://mlsanity.com/paper/2007.08509/) (2020)
- [In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking](https://mlsanity.com/paper/1806.02877/) (2018)
- [RSGAN: Face Swapping and Editing using Face and Hair Representation in Latent Spaces](https://mlsanity.com/paper/1804.03447/) (2018)
- [COCO-FUNIT: Few-Shot Unsupervised Image Translation with a Content Conditioned Style Encoder](https://mlsanity.com/paper/2007.07431/) (2020)
- [Deepfakes Detection with Automatic Face Weighting](https://mlsanity.com/paper/2004.12027/) (2020)

## Built on by (most cited first, 30 of 79)

- [M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection](https://mlsanity.com/paper/2104.09770/) (2021)
- [TALL: Thumbnail Layout for Deepfake Video Detection](https://mlsanity.com/paper/2307.07494/) (2023)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [Combining EfficientNet and Vision Transformers for Video Deepfake Detection](https://mlsanity.com/paper/2107.02612/) (2021)
- [AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection](https://mlsanity.com/paper/2406.02951/) (2024)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Deepfake Detection: A Comprehensive Survey from the Reliability Perspective](https://mlsanity.com/paper/2211.10881/) (2022)
- [Voice-Face Homogeneity Tells Deepfake](https://mlsanity.com/paper/2203.02195/) (2022)
- [Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization](https://mlsanity.com/paper/2305.01979/) (2023)
- [On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection](https://mlsanity.com/paper/2410.23623/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes](https://mlsanity.com/paper/2401.15668/) (2024)
- [MVT: Mask Vision Transformer for Facial Expression Recognition in the wild](https://mlsanity.com/paper/2106.04520/) (2021)
- [MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection](https://mlsanity.com/paper/2211.10996/) (2022)
- [Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content](https://mlsanity.com/paper/2412.12278/) (2024)
- [Learning Spatiotemporal Inconsistency via Thumbnail Layout for Face Deepfake Detection](https://mlsanity.com/paper/2403.10261/) (2024)
- [Deepfake Video Detection with Spatiotemporal Dropout Transformer](https://mlsanity.com/paper/2207.06612/) (2022)
- [Deep Convolutional Pooling Transformer for Deepfake Detection](https://mlsanity.com/paper/2209.05299/) (2022)
- [A Review of Deep Learning-based Approaches for Deepfake Content Detection](https://mlsanity.com/paper/2202.06095/) (2022)
- [GenFace: A Large-Scale Fine-Grained Face Forgery Benchmark and Cross Appearance-Edge Learning](https://mlsanity.com/paper/2402.02003/) (2024)
- [MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection](https://mlsanity.com/paper/2409.09724/) (2024)
- [MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2310.02234/) (2023)
- [GenAI Mirage: The Impostor Bias and the Deepfake Detection Challenge in the Era of Artificial Illusions](https://mlsanity.com/paper/2312.16220/) (2023)
- [A Timely Survey on Vision Transformer for Deepfake Detection](https://mlsanity.com/paper/2405.08463/) (2024)
- [Detection of Deepfake Videos Using Long Distance Attention](https://mlsanity.com/paper/2106.12832/) (2021)
- [AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies](https://mlsanity.com/paper/2403.14974/) (2024)
- [Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection](https://mlsanity.com/paper/2206.13829/) (2022)
- [AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection](https://mlsanity.com/paper/2310.13103/) (2023)
- [Non-Consensual Synthetic Intimate Imagery: Prevalence, Attitudes, and Knowledge in 10 Countries](https://mlsanity.com/paper/2402.01721/) (2024)

## Abstract (arXiv)

The rapid advancement of deep learning models that can generate and synthesis hyper-realistic videos known as Deepfakes and their ease of access to the general public have raised concern from all concerned bodies to their possible malicious intent use. Deep learning techniques can now generate faces, swap faces between two subjects in a video, alter facial expressions, change gender, and alter facial features, to list a few. These powerful video manipulation methods have potential use in many fields. However, they also pose a looming threat to everyone if used for harmful purposes such as identity theft, phishing, and scam. In this work, we propose a Convolutional Vision Transformer for the detection of Deepfakes. The Convolutional Vision Transformer has two components: Convolutional Neural Network (CNN) and Vision Transformer (ViT). The CNN extracts learnable features while the ViT takes in the learned features as input and categorizes them using an attention mechanism. We trained our model on the DeepFake Detection Challenge Dataset (DFDC) and have achieved 91.5 percent accuracy, an AUC value of 0.91, and a loss value of 0.32. Our contribution is that we have added a CNN module to the ViT architecture and have achieved a competitive result on the DFDC dataset.