mlsanityData to 14 Oct 2026

DF-TransFusion: Multimodal Deepfake Detection via Lip-Audio Cross-Attention and Facial Self-Attention

arXiv 2309.06511 ↗First posted 12 Sep 2023cs.CV cs.MMcited by 3

Not on the map. Why?

Abstract (arXiv)

With the rise in manipulated media, deepfake detection has become an imperative task for preserving the authenticity of digital content. In this paper, we present a novel multi-modal audio-video framework designed to concurrently process audio and video inputs for deepfake detection tasks. Our model capitalizes on lip synchronization with input audio through a cross-attention mechanism while extracting visual cues via a fine-tuned VGG-16 network. Subsequently, a transformer encoder network is employed to perform facial self-attention. We conduct multiple ablation studies highlighting different strengths of our approach. Our multi-modal methodology outperforms state-of-the-art multi-modal deepfake detection techniques in terms of F-1 and per-video AUC scores.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,101
  7. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 632
  8. 2017Generative Adversarial Networks: An Overviewcited by 590
  9. 2018MesoNet: a Compact Facial Video Forgery Detection Networkcited by 409
  10. 2021Multi-attentional Deepfake Detectioncited by 341
  11. 2018Exposing DeepFake Videos By Detecting Face Warping Artifactscited by 319
  12. 2018Exposing Deep Fakes Using Inconsistent Head Posescited by 297
20 more
  1. 2020Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detectioncited by 287
  2. 2020DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detectioncited by 254
  3. 2018DeepFakes: a New Threat to Face Recognition? Assessment and Detectioncited by 209
  4. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 181
  5. 2018Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videoscited by 180
  6. 2020Two-branch Recurrent Network for Isolating Deepfakes in Videoscited by 161
  7. 2020DeepFaceLab: Integrated, flexible and extensible face-swapping frameworkcited by 131
  8. 2019Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videoscited by 130
  9. 2018Two-Stream Neural Networks for Tampered Face Detectioncited by 125
  10. 2019Recurrent Convolutional Strategies for Face Manipulation Detection in Videoscited by 118
  11. 2020Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cuescited by 111
  12. 2018Learning to Navigate for Fine-grained Classificationcited by 103
  13. 2020Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localizationcited by 89
  14. 2021Spatiotemporal Inconsistency Learning for DeepFake Video Detectioncited by 88
  15. 2021Combining EfficientNet and Vision Transformers for Video Deepfake Detectioncited by 82
  16. 2018Dynamic Temporal Alignment of Speech to Lipscited by 13
  17. 2022Fully Automated End-to-End Fake Audio Detectioncited by 10
  18. 2019The Liar's Walk: Detecting Deception with Gait and Gesturecited by 3
  19. 2017Contrast Enhancement Estimation for Digital Image Forensicscited by 2
  20. 2020Automatic Speech Verification Spoofing Detectioncited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challengescited by 6
  2. 2025Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localizationcited by 2
  3. 2025Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systemscited by 0