mlsanityData to 7 Oct 2026

Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization

arXiv 2005.14405 ↗First posted 29 May 2020cs.CV cs.MMcited by 89

Not on the map. Why?

Abstract (arXiv)

We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, eg, loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as an aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 13; 2023: 14; 2024: 19; 2025: 20; 2026: 14.

89 papers.

Sort:

By area:

  1. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  2. 2021Face Forensics in the Wildcited by 87
  3. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  4. 2021Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forwardcited by 82
  5. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  6. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  7. 2021Countering Malicious DeepFakes: Survey, Battleground, and Horizoncited by 56
  8. 2021CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakescited by 54
  9. 2024Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detectioncited by 47
  10. 2022Audio-Visual Person-of-Interest DeepFake Detectioncited by 43
  11. 2023Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localizationcited by 42
  12. 2022Exploring Disentangled Content Information for Face Forgery Detectioncited by 36
38 more
  1. 2022Learning Second Order Local Anomaly for General Face Forgery Detectioncited by 36
  2. 2023UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localizationcited by 33
  3. 2021Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectorscited by 32
  4. 2024Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakescited by 32
  5. 2024Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learningcited by 23
  6. 2023Detecting Deepfakes Without Seeing Anycited by 17
  7. 2021Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motioncited by 15
  8. 2024DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localizationcited by 12
  9. 2023MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detectioncited by 10
  10. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
  11. 2024AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategiescited by 9
  12. 2022FakeOut: Leveraging Out-of-domain Self-supervision for Multi-modal Video Deepfake Detectioncited by 8
  13. 2023AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detectioncited by 8
  14. 2023Unsupervised Multimodal Deepfake Detection Using Intra- and Cross-Modal Inconsistenciescited by 8
  15. 2025A Multimodal Deviation Perceiving Framework for Weakly-Supervised Temporal Forgery Localizationcited by 8
  16. 2021FakeBuster: A DeepFakes Detection Tool for Video Conferencing Scenarioscited by 6
  17. 2021Contrastive Learning of Global-Local Video Representationscited by 6
  18. 2022Combining Automatic Speaker Verification and Prosody Analysis for Synthetic Speech Detectioncited by 6
  19. 2024Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localizationcited by 6
  20. 2024Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistenciescited by 6
  21. 2025CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillationcited by 6
  22. 2025SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detectioncited by 6
  23. 2022Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in VIS and NIR Scenariocited by 5
  24. 2023COMICS: End-to-end Bi-grained Contrastive Learning for Multi-face Forgery Detectioncited by 5
  25. 2025PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysiscited by 5
  26. 2022Exploring Spatial-Temporal Features for Deepfake Detection and Localizationcited by 4
  27. 2022Cross-Domain Local Characteristic Enhanced Deepfake Video Detectioncited by 4
  28. 2023Undercover Deepfakes: Detecting Fake Segments in Videoscited by 4
  29. 2023NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake Detectioncited by 4
  30. 20241M-Deepfakes Detection Challengecited by 4
  31. 2024Passive Deepfake Detection Across Multi-modalities: A Comprehensive Surveycited by 4
  32. 2025Audio-Visual Deepfake Detection With Local Temporal Inconsistenciescited by 4
  33. 2023DF-TransFusion: Multimodal Deepfake Detection via Lip-Audio Cross-Attention and Facial Self-Attentioncited by 3
  34. 2024Explicit Correlation Learning for Generalizable Cross-Modal Deepfake Detectioncited by 3
  35. 2024A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detectioncited by 3
  36. 2024Statistics-aware Audio-visual Deepfake Detectorcited by 3
  37. 2025Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learningcited by 3
  38. 2025DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detectioncited by 3

50 most cited of 89 All 89 (JSON)