# UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization

arXiv: https://arxiv.org/abs/2308.14395 · HTML: https://mlsanity.com/paper/2308.14395/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 68 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection](https://mlsanity.com/paper/2101.01456/) (2021)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos](https://mlsanity.com/paper/1810.11215/) (2018)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [DeepFaceLab: Integrated, flexible and extensible face-swapping framework](https://mlsanity.com/paper/2005.05535/) (2020)
- [ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis](https://mlsanity.com/paper/2103.05630/) (2021)
- [Two-Stream Neural Networks for Tampered Face Detection](https://mlsanity.com/paper/1803.11276/) (2018)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)

## Built on by (most cited first, 30 of 33)

- [AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset](https://mlsanity.com/paper/2311.15308/) (2023)
- [A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authentication](https://mlsanity.com/paper/2407.10575/) (2024)
- [DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization](https://mlsanity.com/paper/2411.10193/) (2024)
- [Coarse-to-Fine Proposal Refinement Framework for Audio Temporal Forgery Detection and Localization](https://mlsanity.com/paper/2407.16554/) (2024)
- [A Multimodal Deviation Perceiving Framework for Weakly-Supervised Temporal Forgery Localization](https://mlsanity.com/paper/2507.16596/) (2025)
- [Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges](https://mlsanity.com/paper/2406.06965/) (2024)
- [Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization](https://mlsanity.com/paper/2408.01532/) (2024)
- [Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network](https://mlsanity.com/paper/2505.01880/) (2025)
- [KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features](https://mlsanity.com/paper/2508.07337/) (2025)
- [1M-Deepfakes Detection Challenge](https://mlsanity.com/paper/2409.06991/) (2024)
- [AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization](https://mlsanity.com/paper/2511.18993/) (2025)
- [PhonemeFake: Redefining Deepfake Realism with Language-Driven Segmental Manipulation and Adaptive Bilevel Detection](https://mlsanity.com/paper/2506.22783/) (2025)
- [Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning](https://mlsanity.com/paper/2508.02179/) (2025)
- [Pindrop it! Audio and Visual Deepfake Countermeasures for Robust Detection and Fine Grained-Localization](https://mlsanity.com/paper/2508.08141/) (2025)
- [FakeParts: a New Family of AI-Generated DeepFakes](https://mlsanity.com/paper/2508.21052/) (2025)
- [Context-aware TFL: A Universal Context-aware Contrastive Learning Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2506.08493/) (2025)
- [Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling](https://mlsanity.com/paper/2508.02000/) (2025)
- [Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localization](https://mlsanity.com/paper/2511.10212/) (2025)
- [METER: Multi-modal Evidence-based Thinking and Explainable Reasoning -- Algorithm and Benchmark](https://mlsanity.com/paper/2507.16206/) (2025)
- [ActivityForensics: A Comprehensive Benchmark for Localizing Manipulated Activity in Videos](https://mlsanity.com/paper/2604.03819/) (2026)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [3-Tracer: A Tri-level Temporal-Aware Framework for Audio Forgery Detection and Localization](https://mlsanity.com/paper/2511.21237/) (2025)
- [OmniFD: A Unified Model for Versatile Face Forgery Detection](https://mlsanity.com/paper/2512.01128/) (2025)
- [DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization](https://mlsanity.com/paper/2603.04882/) (2026)
- [GEM-TFL: Bridging Weak and Full Supervision for Forgery Localization through EM-Guided Decomposition and Temporal Refinement](https://mlsanity.com/paper/2603.05095/) (2026)
- [Word-Anchored Temporal Forgery Localization](https://mlsanity.com/paper/2603.06220/) (2026)
- [Towards multi-modal forgery representation learning for AI-generated video detection and localization](https://mlsanity.com/paper/2605.07232/) (2026)
- [Inconsistency-aware Multimodal Schr"odinger Bridge for Deepfake Localization](https://mlsanity.com/paper/2605.23113/) (2026)
- [Detecting Temporally Localized Manipulations in Authentic Video Streams](https://mlsanity.com/paper/2606.07090/) (2026)
- [LoCC: Detection and Localization of Lip-Syncing Deepfakes via Counterfactual Frame Consistency](https://mlsanity.com/paper/2606.22772/) (2026)

## Abstract (arXiv)

The emergence of artificial intelligence-generated content (AIGC) has raised concerns about the authenticity of multimedia content in various fields. However, existing research for forgery content detection has focused mainly on binary classification tasks of complete videos, which has limited applicability in industrial settings. To address this gap, we propose UMMAFormer, a novel universal transformer framework for temporal forgery localization (TFL) that predicts forgery segments with multimodal adaptation. Our approach introduces a Temporal Feature Abnormal Attention (TFAA) module based on temporal feature reconstruction to enhance the detection of temporal differences. We also design a Parallel Cross-Attention Feature Pyramid Network (PCA-FPN) to optimize the Feature Pyramid Network (FPN) for subtle feature enhancement. To evaluate the proposed method, we contribute a novel Temporal Video Inpainting Localization (TVIL) dataset specifically tailored for video inpainting scenes. Our experiments show that our approach achieves state-of-the-art performance on benchmark datasets, including Lav-DF, TVIL, and Psynd, significantly outperforming previous methods. The code and data are available at https://github.com/ymhzyj/UMMAFormer/.