# WWW: Where, Which and Whatever Enhancing Interpretability in Multimodal Deepfake Detection

arXiv: https://arxiv.org/abs/2408.02954 · HTML: https://mlsanity.com/paper/2408.02954/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 17 of 17)

- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection](https://mlsanity.com/paper/2001.00179/) (2020)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection](https://mlsanity.com/paper/2104.09770/) (2021)
- [Deep Learning for Deepfakes Creation and Detection: A Survey](https://mlsanity.com/paper/1909.11573/) (2019)
- [ID-Reveal: Identity-aware DeepFake Video Detection](https://mlsanity.com/paper/2012.02512/) (2020)
- [Spatiotemporal Inconsistency Learning for DeepFake Video Detection](https://mlsanity.com/paper/2109.01860/) (2021)
- [Self-Supervised Video Forensics by Audio-Visual Anomaly Detection](https://mlsanity.com/paper/2301.01767/) (2023)
- [DeepFake Detection by Analyzing Convolutional Traces](https://mlsanity.com/paper/2004.10448/) (2020)
- [KoDF: A Large-scale Korean DeepFake Detection Dataset](https://mlsanity.com/paper/2103.10094/) (2021)
- [Sharp Multiple Instance Learning for DeepFake Video Detection](https://mlsanity.com/paper/2008.04585/) (2020)
- [Video Transformer for Deepfake Detection with Incremental Learning](https://mlsanity.com/paper/2108.05307/) (2021)
- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [Self-supervised Transformer for Deepfake Detection](https://mlsanity.com/paper/2203.01265/) (2022)
- [Extracting deep local features to detect manipulated images of human faces](https://mlsanity.com/paper/1911.13269/) (2019)
- [Using GANs to Synthesise Minimum Training Data for Deepfake Generation](https://mlsanity.com/paper/2011.05421/) (2020)

## Built on by (most cited first, 2 of 2)

- [Detecting AI-Generated Video: A Vision-Language Dual-View Survey](https://mlsanity.com/paper/2607.10787/) (2026)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)

## Abstract (arXiv)

All current benchmarks for multimodal deepfake detection manipulate entire frames using various generation techniques, resulting in oversaturated detection accuracies exceeding 94% at the video-level classification. However, these benchmarks struggle to detect dynamic deepfake attacks with challenging frame-by-frame alterations presented in real-world scenarios. To address this limitation, we introduce FakeMix, a novel clip-level evaluation benchmark aimed at identifying manipulated segments within both video and audio, providing insight into the origins of deepfakes. Furthermore, we propose novel evaluation metrics, Temporal Accuracy (TA) and Frame-wise Discrimination Metric (FDM), to assess the robustness of deepfake detection models. Evaluating state-of-the-art models against diverse deepfake benchmarks, particularly FakeMix, demonstrates the effectiveness of our approach comprehensively. Specifically, while achieving an Average Precision (AP) of 94.2% at the video-level, the evaluation of the existing models at the clip-level using the proposed metrics, TA and FDM, yielded sharp declines in accuracy to 53.1%, and 52.1%, respectively.