mlsanityData to 7 Oct 2026

Towards multi-modal forgery representation learning for AI-generated video detection and localization

arXiv 2605.07232 ↗First posted 8 May 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in generative AI have democratized video creation at scale. AI-generated videos, including partially manipulated clips across visual and audio channels, pose escalating risks of semantic distortion and misuse, which motivates the need for reliable detection tools. Most existing AI-generated video detectors remain limited by single- or partial-modality of data modeling and the lack of fine-grained temporal forgery localization. To address these challenges, our primary novelty introduces a core architecture that jointly integrates an LMM semantic branch with a spatio-temporal (ST) visual branch and a multi-scale partial-spoof (PS) audio branch. This multi-modal approach enables simultaneous detection and fine-grained temporal localization of partially manipulated AI-generated video forgeries. Extensive experiments show that this approach outperforms existing state-of-the-art methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

14 of 17 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021Pay Attention to MLPscited by 367
  6. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 179
  7. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  8. 2024Preserving Fairness Generalization in Deepfake Detectioncited by 70
  9. 2021A Comparative Study on Recent Neural Spoofing Countermeasures for Synthetic Speech Detectioncited by 56
  10. 2022The PartialSpoof Database and Countermeasures for the Detection of Short Fake Speech Segments Embedded in an Utterancecited by 47
  11. 2024On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detectioncited by 40
  12. 2023UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localizationcited by 33
2 more
  1. 2019P2SGrad: Refined Gradients for Optimizing Deep Face Modelscited by 14
  2. 2025AV-Deepfake1M++: A Large-Scale Audio-Visual Deepfake Benchmark with Real-World Perturbationscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization
  2. 2026UniSkip-Mamba: A Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization
  3. 2025Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning
  4. 2025Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning
  5. 2026ManiVid: Unified and Explainable Forensic Analysis of Manipulated Videos
  6. 2025METER: Multi-modal Evidence-based Thinking and Explainable Reasoning -- Algorithm and Benchmark
  7. 2025Can GPT tell us why these images are synthesized? Empowering Multimodal Large Language Models for Forensics
  8. 2025BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos