mlsanityData to 14 Oct 2026

Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization

arXiv 2204.06228 ↗First posted 13 Apr 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Due to its high societal impact, deepfake detection is getting active attention in the computer vision community. Most deepfake detection methods rely on identity, facial attributes, and adversarial perturbation-based spatio-temporal modifications at the whole video or random locations while keeping the meaning of the content intact. However, a sophisticated deepfake may contain only a small segment of video/audio manipulation, through which the meaning of the content can be, for example, completely inverted from a sentiment perspective. We introduce a content-driven audio-visual deepfake dataset, termed Localized Audio Visual DeepFake (LAV-DF), explicitly designed for the task of learning temporal forgery localization. Specifically, the content-driven audio-visual manipulations are performed strategically to change the sentiment polarity of the whole video. Our baseline method for benchmarking the proposed dataset is a 3DCNN model, termed as Boundary Aware Temporal Forgery Detection (BA-TFD), which is guided via contrastive, boundary matching, and frame classification loss functions. Our extensive quantitative and qualitative analysis demonstrates the proposed method's strong performance for temporal forgery localization and deepfake detection tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 58 of its 76 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

58 of 76 references.

References · 58

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,101
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  8. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  9. 2019Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensicscited by 575
  10. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  11. 2020Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Cluescited by 523
  12. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
38 more
  1. 2018MesoNet: a Compact Facial Video Forgery Detection Networkcited by 409
  2. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 382
  3. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  4. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 329
  5. 2019FSGAN: Subject Agnostic Face Swapping and Reenactmentcited by 320
  6. 2018Exposing DeepFake Videos By Detecting Face Warping Artifactscited by 319
  7. 2018Exposing Deep Fakes Using Inconsistent Head Posescited by 297
  8. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  9. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  10. 2017Temporal Action Detection with Structured Segment Networkscited by 267
  11. 2020DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detectioncited by 254
  12. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  13. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  14. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 249
  15. 2020The Creation and Detection of Deepfakes: A Surveycited by 247
  16. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  17. 2018DeepFakes: a New Threat to Face Recognition? Assessment and Detectioncited by 209
  18. 2021WildDeepfake: A Challenging Real-World Dataset for Deepfake Detectioncited by 209
  19. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  20. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 181
  21. 2021ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysiscited by 126
  22. 2017Single Shot Temporal Action Detectioncited by 123
  23. 2016Fast Face-swap Using Convolutional Neural Networkscited by 119
  24. 2021M2TR: Multi-modal Multi-scale Transformers for Deepfake Detectioncited by 114
  25. 2017TURN TAP: Temporal Unit Regression Network for Temporal Action Proposalscited by 113
  26. 2017CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videoscited by 112
  27. 2020Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cuescited by 111
  28. 2020Towards Automatic Face-to-Face Translationcited by 96
  29. 2020Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localizationcited by 89
  30. 2021Spatiotemporal Inconsistency Learning for DeepFake Video Detectioncited by 88
  31. 2021Combining EfficientNet and Vision Transformers for Video Deepfake Detectioncited by 82
  32. 2020DeepFake Detection by Analyzing Convolutional Tracescited by 81
  33. 2021Deepfake Video Detection Using Convolutional Vision Transformercited by 79
  34. 2020BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generationcited by 65
  35. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
  36. 2020Multi-shot Temporal Event Localization: a Benchmarkcited by 52
  37. 2021OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery Detection And Segmentation In-The-Wildcited by 50
  38. 2021Activity Graph Transformer for Temporal Action Localizationcited by 48

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.