# TAR: Generalized Forensic Framework to Detect Deepfakes using Weakly Supervised Learning

arXiv: https://arxiv.org/abs/2105.06117 · HTML: https://mlsanity.com/paper/2105.06117/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 13 May 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [Learning Rich Features for Image Manipulation Detection](https://mlsanity.com/paper/1805.04953/) (2018)
- [FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals](https://mlsanity.com/paper/1901.02212/) (2019)
- [Fighting Fake News: Image Splice Detection via Learned Self-Consistency](https://mlsanity.com/paper/1805.04096/) (2018)
- [Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos](https://mlsanity.com/paper/1906.06876/) (2019)
- [ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection](https://mlsanity.com/paper/1812.02510/) (2018)
- [Image Splicing Localization Using A Multi-Task Fully Convolutional Network (MFCN)](https://mlsanity.com/paper/1709.02016/) (2017)
- [Recasting Residual-based Local Descriptors as Convolutional Neural Networks: an Application to Image Forgery Detection](https://mlsanity.com/paper/1703.04615/) (2017)
- [One Detector to Rule Them All: Towards a General Deepfake Attack Detection Framework](https://mlsanity.com/paper/2105.00187/) (2021)
- [A Convolutional LSTM based Residual Network for Deepfake Video Detection](https://mlsanity.com/paper/2009.07480/) (2020)
- [Am I a Real or Fake Celebrity? Measuring Commercial Face Recognition Web APIs under Deepfake Impersonation Attack](https://mlsanity.com/paper/2103.00847/) (2021)

## Built on by (most cited first, 19 of 19)

- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning](https://mlsanity.com/paper/2105.13617/) (2021)
- [CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation](https://mlsanity.com/paper/2107.02408/) (2021)
- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework](https://mlsanity.com/paper/2401.04364/) (2024)
- [Why Do Facial Deepfake Detectors Fail?](https://mlsanity.com/paper/2302.13156/) (2023)
- [Learning Pairwise Interaction for Generalizable DeepFake Detection](https://mlsanity.com/paper/2302.13288/) (2023)
- [GenAI Mirage: The Impostor Bias and the Deepfake Detection Challenge in the Era of Artificial Illusions](https://mlsanity.com/paper/2312.16220/) (2023)
- [Deepfake in the Metaverse: Security Implications for Virtual Gaming, Meetings, and Offices](https://mlsanity.com/paper/2303.14612/) (2023)
- [UGAD: Universal Generative AI Detector utilizing Frequency Fingerprints](https://mlsanity.com/paper/2409.07913/) (2024)
- [Exploring the Impact of Moire Pattern on Deepfake Detectors](https://mlsanity.com/paper/2407.10399/) (2024)
- [How Generalizable are Deepfake Image Detectors? An Empirical Study](https://mlsanity.com/paper/2308.04177/) (2023)
- [FakeChain: Exposing Shallow Cues in Multi-Step Deepfake Detection](https://mlsanity.com/paper/2509.16602/) (2025)
- [From Prediction to Explanation: Multimodal, Explainable, and Interactive Deepfake Detection Framework for Non-Expert Users](https://mlsanity.com/paper/2508.07596/) (2025)
- [Seeing Through the Blur: Unlocking Defocus Maps for Deepfake Detection](https://mlsanity.com/paper/2509.23289/) (2025)
- [Deepfake Detection for Facial Images with Facemasks](https://mlsanity.com/paper/2202.11359/) (2022)
- [Beyond Deepfake vs Real: Facial Deepfake Detection in the Open-Set Paradigm](https://mlsanity.com/paper/2503.08055/) (2025)
- [Through the Lens: Benchmarking Deepfake Detectors Against Moir'e-Induced Distortions](https://mlsanity.com/paper/2510.23225/) (2025)
- [Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection](https://mlsanity.com/paper/2606.15117/) (2026)

## Abstract (arXiv)

Deepfakes have become a critical social problem, and detecting them is of utmost importance. Also, deepfake generation methods are advancing, and it is becoming harder to detect. While many deepfake detection models can detect different types of deepfakes separately, they perform poorly on generalizing the detection performance over multiple types of deepfake. This motivates us to develop a generalized model to detect different types of deepfakes. Therefore, in this work, we introduce a practical digital forensic tool to detect different types of deepfakes simultaneously and propose Transfer learning-based Autoencoder with Residuals (TAR). The ultimate goal of our work is to develop a unified model to detect various types of deepfake videos with high accuracy, with only a small number of training samples that can work well in real-world settings. We develop an autoencoder-based detection model with Residual blocks and sequentially perform transfer learning to detect different types of deepfakes simultaneously. Our approach achieves a much higher generalized detection performance than the state-of-the-art methods on the FaceForensics++ dataset. In addition, we evaluate our model on 200 real-world Deepfake-in-the-Wild (DW) videos of 50 celebrities available on the Internet and achieve 89.49% zero-shot accuracy, which is significantly higher than the best baseline model (gaining 10.77%), demonstrating and validating the practicability of our approach.