# FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning

arXiv: https://arxiv.org/abs/2105.13617 · HTML: https://mlsanity.com/paper/2105.13617/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 49 of its 91 references that are in the map.
- First posted on arXiv 28 May 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 49)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Exposing DeepFake Videos By Detecting Face Warping Artifacts](https://mlsanity.com/paper/1811.00656/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Exposing Deep Fakes Using Inconsistent Head Poses](https://mlsanity.com/paper/1811.00661/) (2018)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [Learning Rich Features for Image Manipulation Detection](https://mlsanity.com/paper/1805.04953/) (2018)
- [Fighting Fake News: Image Splice Detection via Learned Self-Consistency](https://mlsanity.com/paper/1805.04096/) (2018)
- [Two-Stream Neural Networks for Tampered Face Detection](https://mlsanity.com/paper/1803.11276/) (2018)
- [Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN](https://mlsanity.com/paper/1811.09245/) (2018)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)

## Built on by (most cited first, 30 of 44)

- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Fusing Global and Local Features for Generalized AI-Synthesized Image Detection](https://mlsanity.com/paper/2203.13964/) (2022)
- [SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes](https://mlsanity.com/paper/2211.11296/) (2022)
- [CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation](https://mlsanity.com/paper/2107.02408/) (2021)
- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [GLFF: Global and Local Feature Fusion for AI-synthesized Image Detection](https://mlsanity.com/paper/2211.08615/) (2022)
- [Discovering Transferable Forensic Features for CNN-generated Images Detection](https://mlsanity.com/paper/2208.11342/) (2022)
- [Continual Face Forgery Detection via Historical Distribution Preserving](https://mlsanity.com/paper/2308.06217/) (2023)
- [Controllable Guide-Space for Generalizable Face Forgery Detection](https://mlsanity.com/paper/2307.14039/) (2023)
- [Model Attribution of Face-swap Deepfake Videos](https://mlsanity.com/paper/2202.12951/) (2022)
- [SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework](https://mlsanity.com/paper/2401.04364/) (2024)
- [A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authentication](https://mlsanity.com/paper/2407.10575/) (2024)
- [Why Do Facial Deepfake Detectors Fail?](https://mlsanity.com/paper/2302.13156/) (2023)
- [Learning Pairwise Interaction for Generalizable DeepFake Detection](https://mlsanity.com/paper/2302.13288/) (2023)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [Harmfully Manipulated Images Matter in Multimodal Misinformation Detection](https://mlsanity.com/paper/2407.19192/) (2024)
- [Deepfake in the Metaverse: Security Implications for Virtual Gaming, Meetings, and Offices](https://mlsanity.com/paper/2303.14612/) (2023)
- [Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues](https://mlsanity.com/paper/2404.08450/) (2024)
- [The MeVer DeepFake Detection Service: Lessons Learnt from Developing and Deploying in the Wild](https://mlsanity.com/paper/2204.12816/) (2022)
- [Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges](https://mlsanity.com/paper/2406.06965/) (2024)
- [VRM: Knowledge Distillation via Virtual Relation Matching](https://mlsanity.com/paper/2502.20760/) (2025)
- [Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in VIS and NIR Scenario](https://mlsanity.com/paper/2207.01906/) (2022)
- [Dual-Branch Knowledge Distillation for Noise-Robust Synthetic Speech Detection](https://mlsanity.com/paper/2310.08869/) (2023)
- [UGAD: Universal Generative AI Detector utilizing Frequency Fingerprints](https://mlsanity.com/paper/2409.07913/) (2024)
- [Assessment Framework for Deepfake Detection in Real-world Situations](https://mlsanity.com/paper/2304.06125/) (2023)
- [Undercover Deepfakes: Detecting Fake Segments in Videos](https://mlsanity.com/paper/2305.06564/) (2023)
- [Exploring the Impact of Moire Pattern on Deepfake Detectors](https://mlsanity.com/paper/2407.10399/) (2024)
- [LLMs Are Not Yet Ready for Deepfake Image Detection](https://mlsanity.com/paper/2506.10474/) (2025)
- [Quantitative Metrics for Evaluating Explanations of Video DeepFake Detectors](https://mlsanity.com/paper/2210.03683/) (2022)
- [How Generalizable are Deepfake Image Detectors? An Empirical Study](https://mlsanity.com/paper/2308.04177/) (2023)

## Abstract (arXiv)

As GAN-based video and image manipulation technologies become more sophisticated and easily accessible, there is an urgent need for effective deepfake detection technologies. Moreover, various deepfake generation techniques have emerged over the past few years. While many deepfake detection methods have been proposed, their performance suffers from new types of deepfake methods on which they are not sufficiently trained. To detect new types of deepfakes, the model should learn from additional data without losing its prior knowledge about deepfakes (catastrophic forgetting), especially when new deepfakes are significantly different. In this work, we employ the Representation Learning (ReL) and Knowledge Distillation (KD) paradigms to introduce a transfer learning-based Feature Representation Transfer Adaptation Learning (FReTAL) method. We use FReTAL to perform domain adaptation tasks on new deepfake datasets while minimizing catastrophic forgetting. Our student model can quickly adapt to new types of deepfake by distilling knowledge from a pre-trained teacher model and applying transfer learning without using source domain data during domain adaptation. Through experiments on FaceForensics++ datasets, we demonstrate that FReTAL outperforms all baselines on the domain adaptation task with up to 86.97% accuracy on low-quality deepfakes.