# UIA-ViT: Unsupervised Inconsistency-Aware Method based on Vision Transformer for Face Forgery Detection

arXiv: https://arxiv.org/abs/2210.12752 · HTML: https://mlsanity.com/paper/2210.12752/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [On the Detection of Digital Face Manipulation](https://mlsanity.com/paper/1910.01717/) (2019)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [Learning Self-Consistency for Deepfake Detection](https://mlsanity.com/paper/2012.09311/) (2020)
- [Two-branch Recurrent Network for Isolating Deepfakes in Videos](https://mlsanity.com/paper/2008.03412/) (2020)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection](https://mlsanity.com/paper/2103.09096/) (2021)
- [Modeling the Distribution of Normal Data in Pre-Trained Deep Features for Anomaly Detection](https://mlsanity.com/paper/2005.14140/) (2020)
- [Local Relation Learning for Face Forgery Detection](https://mlsanity.com/paper/2105.02577/) (2021)
- [Dual Contrastive Learning for General Face Forgery Detection](https://mlsanity.com/paper/2112.13522/) (2021)
- [Video Transformer for Deepfake Detection with Incremental Learning](https://mlsanity.com/paper/2108.05307/) (2021)
- [Towards Generalizable and Robust Face Manipulation Detection via Bag-of-local-feature](https://mlsanity.com/paper/2103.07915/) (2021)

## Built on by (most cited first, 30 of 69)

- [AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection](https://mlsanity.com/paper/2406.02951/) (2024)
- [Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning](https://mlsanity.com/paper/2408.17065/) (2024)
- [Beyond the Prior Forgery Knowledge: Mining Critical Clues for General Face Forgery Detection](https://mlsanity.com/paper/2304.12489/) (2023)
- [SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes](https://mlsanity.com/paper/2211.11296/) (2022)
- [AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics](https://mlsanity.com/paper/2304.06870/) (2023)
- [Towards General Visual-Linguistic Face Forgery Detection](https://mlsanity.com/paper/2307.16545/) (2023)
- [X2-DFD: A framework for eXplainable and eXtendable Deepfake Detection](https://mlsanity.com/paper/2410.06126/) (2024)
- [Fake It till You Make It: Curricular Dynamic Forgery Augmentations towards General Deepfake Detection](https://mlsanity.com/paper/2409.14444/) (2024)
- [DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion](https://mlsanity.com/paper/2410.04372/) (2024)
- [MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection](https://mlsanity.com/paper/2404.08452/) (2024)
- [Locate and Verify: A Two-Stream Network for Improved Deepfake Detection](https://mlsanity.com/paper/2309.11131/) (2023)
- [Continual Face Forgery Detection via Historical Distribution Preserving](https://mlsanity.com/paper/2308.06217/) (2023)
- [DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues](https://mlsanity.com/paper/2309.09526/) (2023)
- [FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning](https://mlsanity.com/paper/2412.12032/) (2024)
- [ForensicHub: A Unified Benchmark & Codebase for All-Domain Fake Image Detection and Localization](https://mlsanity.com/paper/2505.11003/) (2025)
- [Controllable Guide-Space for Generalizable Face Forgery Detection](https://mlsanity.com/paper/2307.14039/) (2023)
- [Detecting Deepfakes Without Seeing Any](https://mlsanity.com/paper/2311.01458/) (2023)
- [Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation](https://mlsanity.com/paper/2309.09667/) (2023)
- [SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework](https://mlsanity.com/paper/2401.04364/) (2024)
- [From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakes](https://mlsanity.com/paper/2504.04827/) (2025)
- [A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authentication](https://mlsanity.com/paper/2407.10575/) (2024)
- [Multi-spectral Class Center Network for Face Manipulation Detection and Localization](https://mlsanity.com/paper/2305.10794/) (2023)
- [Learning to Discover Forgery Cues for Face Forgery Detection](https://mlsanity.com/paper/2409.01030/) (2024)
- [MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection](https://mlsanity.com/paper/2409.09724/) (2024)
- [A Timely Survey on Vision Transformer for Deepfake Detection](https://mlsanity.com/paper/2405.08463/) (2024)
- [Real Face Foundation Representation Learning for Generalized Deepfake Detection](https://mlsanity.com/paper/2303.08439/) (2023)
- [Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues](https://mlsanity.com/paper/2404.08450/) (2024)
- [ED4: Explicit Data-level Debiasing for Deepfake Detection](https://mlsanity.com/paper/2408.06779/) (2024)
- [Discrepancy-Guided Reconstruction Learning for Image Forgery Detection](https://mlsanity.com/paper/2304.13349/) (2023)
- [Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding](https://mlsanity.com/paper/2309.12657/) (2023)

## Abstract (arXiv)

Intra-frame inconsistency has been proved to be effective for the generalization of face forgery detection. However, learning to focus on these inconsistency requires extra pixel-level forged location annotations. Acquiring such annotations is non-trivial. Some existing methods generate large-scale synthesized data with location annotations, which is only composed of real images and cannot capture the properties of forgery regions. Others generate forgery location labels by subtracting paired real and fake images, yet such paired data is difficult to collected and the generated label is usually discontinuous. To overcome these limitations, we propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT, which only makes use of video-level labels and can learn inconsistency-aware feature without pixel-level annotations. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning. Based on vision Transformer, we propose two key components: Unsupervised Patch Consistency Learning (UPCL) and Progressive Consistency Weighted Assemble (PCWA). UPCL is designed for learning the consistency-related representation with progressive optimized pseudo annotations. PCWA enhances the final classification embedding with previous patch embeddings optimized by UPCL to further improve the detection performance. Extensive experiments demonstrate the effectiveness of the proposed method.