# Stacking Brick by Brick: Aligned Feature Isolation for Incremental Face Forgery Detection

arXiv: https://arxiv.org/abs/2411.11396 · HTML: https://mlsanity.com/paper/2411.11396/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Builds on (most cited first, 30 of 36)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Detecting Deepfakes with Self-Blended Images](https://mlsanity.com/paper/2204.08376/) (2022)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain](https://mlsanity.com/paper/2103.01856/) (2021)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [UCF: Uncovering Common Features for Generalizable Deepfake Detection](https://mlsanity.com/paper/2304.13949/) (2023)
- [DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection](https://mlsanity.com/paper/2307.01426/) (2023)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [DF40: Toward Next-Generation Deepfake Detection](https://mlsanity.com/paper/2406.13495/) (2024)
- [Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection](https://mlsanity.com/paper/2311.11278/) (2023)
- [Dual Contrastive Learning for General Face Forgery Detection](https://mlsanity.com/paper/2112.13522/) (2021)
- [CORE: Consistent Representation Learning for Face Forgery Detection](https://mlsanity.com/paper/2206.02749/) (2022)
- [BlendFace: Re-designing Identity Encoders for Face-Swapping](https://mlsanity.com/paper/2307.10854/) (2023)
- [Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation](https://mlsanity.com/paper/2307.09906/) (2023)
- [Can We Leave Deepfake Data Behind in Training Deepfake Detector?](https://mlsanity.com/paper/2408.17052/) (2024)
- [Towards Deep Symbolic Reinforcement Learning](https://mlsanity.com/paper/1609.05518/) (2016)

## Built on by (most cited first, 16 of 16)

- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [DeepShield: Fortifying Deepfake Video Detection with Local and Global Forgery Analysis](https://mlsanity.com/paper/2510.25237/) (2025)
- [SAIDO: Generalizable Detection of AI-Generated Images via Scene-Aware and Importance-Guided Dynamic Optimization in Continual Learning](https://mlsanity.com/paper/2512.00539/) (2025)
- [DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces](https://mlsanity.com/paper/2509.19230/) (2025)
- [A Sanity Check for Multi-In-Domain Face Forgery Detection in the Real World](https://mlsanity.com/paper/2512.04837/) (2025)
- [SELFI: Selective Fusion of Identity for Generalizable Deepfake Detection](https://mlsanity.com/paper/2506.17592/) (2025)
- [AIFIND: Artifact-Aware Interpreting Fine-Grained Alignment for Incremental Face Forgery Detection](https://mlsanity.com/paper/2604.16207/) (2026)
- [SphereVideo: Prototype-anchored Hyperspherical Boundary for Continual AI-generated Video Detection](https://mlsanity.com/paper/2608.01334/) (2026)
- [Towards General Visual-Linguistic Face Forgery Detection(V2)](https://mlsanity.com/paper/2502.20698/) (2025)
- [Spoofing-aware Prompt Learning for Unified Physical-Digital Facial Attack Detection](https://mlsanity.com/paper/2512.06363/) (2025)
- [Towards Generalizable Deepfake Detection via Real Distribution Bias Correction](https://mlsanity.com/paper/2603.14005/) (2026)
- [Tutor-Student Reinforcement Learning: A Dynamic Curriculum for Robust Deepfake Detection](https://mlsanity.com/paper/2603.24139/) (2026)
- [Direct Discrepancy Replay: Distribution-Discrepancy Condensation and Manifold-Consistent Replay for Continual Face Forgery Detection](https://mlsanity.com/paper/2604.12941/) (2026)
- [DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts](https://mlsanity.com/paper/2608.23363/) (2026)
- [Preserving Knowledge across Space and Time for Continual Video Deepfake Detection](https://mlsanity.com/paper/2609.03446/) (2026)
- [Forensic-Aware Continual Adaptation for Image Forgery Localization](https://mlsanity.com/paper/2609.38251/) (2026)

## Abstract (arXiv)

The rapid advancement of face forgery techniques has introduced a growing variety of forgeries. Incremental Face Forgery Detection (IFFD), involving gradually adding new forgery data to fine-tune the previously trained model, has been introduced as a promising strategy to deal with evolving forgery methods. However, a naively trained IFFD model is prone to catastrophic forgetting when new forgeries are integrated, as treating all forgeries as a single ''Fake" class in the Real/Fake classification can cause different forgery types overriding one another, thereby resulting in the forgetting of unique characteristics from earlier tasks and limiting the model's effectiveness in learning forgery specificity and generality. In this paper, we propose to stack the latent feature distributions of previous and new tasks brick by brick, textiti.e., achieving aligned feature isolation. In this manner, we aim to preserve learned forgery information and accumulate new knowledge by minimizing distribution overriding, thereby mitigating catastrophic forgetting. To achieve this, we first introduce Sparse Uniform Replay (SUR) to obtain the representative subsets that could be treated as the uniformly sparse versions of the previous global distributions. We then propose a Latent-space Incremental Detector (LID) that leverages SUR data to isolate and align distributions. For evaluation, we construct a more advanced and comprehensive benchmark tailored for IFFD. The leading experimental results validate the superiority of our method.