# On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection

arXiv: https://arxiv.org/abs/2410.23623 · HTML: https://mlsanity.com/paper/2410.23623/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 65 of its 75 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 65)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [DIRE for Diffusion-Generated Image Detection](https://mlsanity.com/paper/2303.09295/) (2023)
- [Multi-attentional Deepfake Detection](https://mlsanity.com/paper/2103.02406/) (2021)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)

## Built on by (most cited first, 30 of 40)

- [Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector](https://mlsanity.com/paper/2503.20188/) (2025)
- [BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanation](https://mlsanity.com/paper/2505.12620/) (2025)
- [AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models](https://mlsanity.com/paper/2507.02664/) (2025)
- [BusterX++: Towards Unified Cross-Modal AI-Generated Content Detection and Explanation with MLLM](https://mlsanity.com/paper/2507.14632/) (2025)
- [Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation](https://mlsanity.com/paper/2506.16802/) (2025)
- [Survey on AI-Generated Media Detection: From Non-MLLM to MLLM](https://mlsanity.com/paper/2502.05240/) (2025)
- [TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection](https://mlsanity.com/paper/2505.24866/) (2025)
- [VideoVeritas: AI-Generated Video Detection via Perception Pretext Reinforcement Learning](https://mlsanity.com/paper/2602.08828/) (2026)
- [Preserving Forgery Artifacts: AI-Generated Video Detection at Native Scale](https://mlsanity.com/paper/2604.04634/) (2026)
- [AvatarShield: Visual Reinforcement Learning for Human-Centric Synthetic Video Detection](https://mlsanity.com/paper/2505.15173/) (2025)
- [AEGIS: Authenticity Evaluation Benchmark for AI-Generated Video Sequences](https://mlsanity.com/paper/2508.10771/) (2025)
- [Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities](https://mlsanity.com/paper/2604.07763/) (2026)
- [Mixture-of-Attack-Experts with Class Regularization for Unified Physical-Digital Face Attack Detection](https://mlsanity.com/paper/2504.00458/) (2025)
- [On the Holistic Approach for Detecting Human Image Forgery](https://mlsanity.com/paper/2601.04715/) (2026)
- [HumanSAM: Classifying Human-centric Forgery Videos in Human Spatial, Appearance, and Motion Anomaly](https://mlsanity.com/paper/2507.19924/) (2025)
- [TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection](https://mlsanity.com/paper/2512.10652/) (2025)
- [FakeParts: a New Family of AI-Generated DeepFakes](https://mlsanity.com/paper/2508.21052/) (2025)
- [MVAD: A Benchmark Dataset for Multimodal AI-Generated Video-Audio Detection](https://mlsanity.com/paper/2512.00336/) (2025)
- [RobustSora: De-Watermarked Benchmark for Robust AI-Generated Video Detection](https://mlsanity.com/paper/2512.10248/) (2025)
- [SAGA: Source Attribution of Generative AI Videos](https://mlsanity.com/paper/2511.12834/) (2025)
- [SafeGuard: A Multi-Agent Perception-Reasoning Framework for Social-Risk AI-Generated Video Detection](https://mlsanity.com/paper/2607.03069/) (2026)
- [EvolveReason: Self-Evolving Reasoning Paradigm for Explainable Deepfake Facial Image Identification](https://mlsanity.com/paper/2603.07515/) (2026)
- [SWIFT: Sliding Window Reconstruction for Few-Shot Training-Free Generated Video Attribution](https://mlsanity.com/paper/2603.08536/) (2026)
- [GenVideoLens: Where LVLMs Fall Short in AI-Generated Video Detection?](https://mlsanity.com/paper/2603.18625/) (2026)
- [Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection](https://mlsanity.com/paper/2607.00948/) (2026)
- [Detecting AI-Generated Video: A Vision-Language Dual-View Survey](https://mlsanity.com/paper/2607.10787/) (2026)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [Dual-Branch Convolutional Framework for Spatial and Frequency-Based Image Forgery Detection](https://mlsanity.com/paper/2509.05281/) (2025)
- [Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning](https://mlsanity.com/paper/2511.18104/) (2025)
- [EA-Swin: An Embedding-Agnostic Swin Transformer for AI-Generated Video Detection](https://mlsanity.com/paper/2602.17260/) (2026)

## Abstract (arXiv)

Large numbers of synthesized videos from diffusion models pose threats to information security and authenticity, leading to an increasing demand for generated content detection. However, existing video-level detection algorithms primarily focus on detecting facial forgeries and often fail to identify diffusion-generated content with a diverse range of semantics. To advance the field of video forensics, we propose an innovative algorithm named Multi-Modal Detection(MM-Det) for detecting diffusion-generated videos. MM-Det utilizes the profound perceptual and comprehensive abilities of Large Multi-modal Models (LMMs) by generating a Multi-Modal Forgery Representation (MMFR) from LMM's multi-modal space, enhancing its ability to detect unseen forgery content. Besides, MM-Det leverages an In-and-Across Frame Attention (IAFA) mechanism for feature augmentation in the spatio-temporal domain. A dynamic fusion strategy helps refine forgery representations for the fusion. Moreover, we construct a comprehensive diffusion video dataset, called Diffusion Video Forensics (DVF), across a wide range of forgery videos. MM-Det achieves state-of-the-art performance in DVF, demonstrating the effectiveness of our algorithm. Both source code and DVF are available at https://github.com/SparkleXFantasy/MM-Det.