mlsanityData to 7 Oct 2026

Self-Supervised Video Forensics by Audio-Visual Anomaly Detection

arXiv 2301.01767 ↗First posted 4 Jan 2023cs.CVcited by 84

Not on the map. Why?

Abstract (arXiv)

Manipulated videos often contain subtle inconsistencies between their visual and audio signals. We propose a video forensics method, based on anomaly detection, that can identify these inconsistencies, and that can be trained solely using real, unlabeled data. We train an autoregressive model to generate sequences of audio-visual features, using feature sets that capture the temporal synchronization between video frames and sound. At test time, we then flag videos that the model assigns low probability. Despite being trained entirely on real videos, our model obtains strong performance on the task of detecting manipulated speech videos. Project site: https://cfeng16.github.io/audio-visual-forensics

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 95 of its 117 references that are in the map. First posted on arXiv 4 Jan 2023.

Builds on

95 of 117 references.

References · 95

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  3. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,095
  6. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  8. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  9. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 842
  10. 2016Pixel Recurrent Neural Networkscited by 814
  11. 2020First Order Motion Model for Image Animationcited by 698
  12. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  13. 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
  14. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  15. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  16. 2020Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Cluescited by 518
  17. 2020Leveraging Frequency Analysis for Deep Fake Image Recognitioncited by 501
  18. 2019Face X-ray for More General Face Forgery Detectioncited by 469
  19. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  20. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  21. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  22. 2019FSGAN: Subject Agnostic Face Swapping and Reenactmentcited by 319
  23. 2018Deep Audio-Visual Speech Recognitioncited by 318
  24. 2018Exposing DeepFake Videos By Detecting Face Warping Artifactscited by 318
  25. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  26. 2019Deep Semi-Supervised Anomaly Detectioncited by 290
  27. 2020Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detectioncited by 286
  28. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  29. 2020Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributionscited by 278
  30. 2020What makes fake images detectable? Understanding properties that generalizecited by 272
  31. 2017Objects that Soundcited by 254
  32. 2020DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detectioncited by 251
  33. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  34. 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 243
  35. 2020Global Texture Enhancement for Fake Face Detection in the Wildcited by 220
  36. 2018Adversarially Learned One-Class Classifier for Novelty Detectioncited by 202
  37. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  38. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 31; 2025: 23; 2026: 21.

84 papers.

Sort:

By area:

  1. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  2. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  3. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  4. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  5. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  6. 2025Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024cited by 40
  7. 2023Generating Visual Scenes from Touchcited by 26
  8. 2024Learning Natural Consistency Representation for Face Forgery Video Detectioncited by 26
  9. 2024Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learningcited by 23
  10. 2024FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learningcited by 21
  11. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  12. 2023Detecting Deepfakes Without Seeing Anycited by 17
38 more
  1. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  2. 2024DeCLIP: Decoding CLIP representations for deepfake localizationcited by 17
  3. 2024Exposing Lip-syncing Deepfakes from Mouth Inconsistenciescited by 15
  4. 2024SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Frameworkcited by 14
  5. 2025From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakescited by 14
  6. 2024NeuroBind: Towards Unified Multimodal Representations for Neural Signalscited by 13
  7. 2025Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Modelscited by 13
  8. 2024A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authenticationcited by 12
  9. 2024DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localizationcited by 12
  10. 2023MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detectioncited by 10
  11. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
  12. 2023Integrating Audio-Visual Features for Multimodal Deepfake Detectioncited by 9
  13. 2023Unsupervised Multimodal Deepfake Detection Using Intra- and Cross-Modal Inconsistenciescited by 8
  14. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  15. 2025Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistenciescited by 7
  16. 2025Beyond Face Swapping: A Diffusion-Based Digital Human Benchmark for Multimodal Deepfake Detectioncited by 7
  17. 2024Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challengescited by 6
  18. 2024Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localizationcited by 6
  19. 2024Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistenciescited by 6
  20. 2025CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillationcited by 6
  21. 2025SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detectioncited by 6
  22. 2025PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysiscited by 5
  23. 2024Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approachcited by 4
  24. 20241M-Deepfakes Detection Challengecited by 4
  25. 2024Passive Deepfake Detection Across Multi-modalities: A Comprehensive Surveycited by 4
  26. 2025Audio-Visual Deepfake Detection With Local Temporal Inconsistenciescited by 4
  27. 2023How Generalizable are Deepfake Image Detectors? An Empirical Studycited by 3
  28. 2024Explicit Correlation Learning for Generalizable Cross-Modal Deepfake Detectioncited by 3
  29. 2024Statistics-aware Audio-visual Deepfake Detectorcited by 3
  30. 2024Toward Robust Real-World Audio Deepfake Detection: Closing the Explainability Gapcited by 3
  31. 2025LLMs Are Not Yet Ready for Deepfake Image Detectioncited by 3
  32. 2025DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detectioncited by 3
  33. 2023AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videoscited by 2
  34. 2024Principles of Designing Robust Remote Face Anti-Spoofing Systemscited by 2
  35. 2024WWW: Where, Which and Whatever Enhancing Interpretability in Multimodal Deepfake Detectioncited by 2
  36. 2024GLCF: A Global-Local Multimodal Coherence Analysis Framework for Talking Face Generation Detectioncited by 2
  37. 2025Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localizationcited by 2
  38. 2026CMTA: Leveraging Cross-Modal Temporal Artifacts for Generalizable AI-Generated Video Detectioncited by 2

50 most cited of 84 All 84 (JSON)