# Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection

arXiv: https://arxiv.org/abs/2401.05746 · HTML: https://mlsanity.com/paper/2401.05746/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2024.

## Built on by (most cited first, 24 of 24)

- [FIRE: Robust Detection of Diffusion-Generated Images via Frequency-Guided Reconstruction Error](https://mlsanity.com/paper/2412.07140/) (2024)
- [DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization](https://mlsanity.com/paper/2411.10193/) (2024)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark](https://mlsanity.com/paper/2505.11109/) (2025)
- [Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges](https://mlsanity.com/paper/2406.06965/) (2024)
- [A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2406.14176/) (2024)
- [DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection](https://mlsanity.com/paper/2510.22622/) (2025)
- [Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localization](https://mlsanity.com/paper/2511.10212/) (2025)
- [CapsFake: A Multimodal Capsule Network for Detecting Instruction-Guided Deepfakes](https://mlsanity.com/paper/2504.19212/) (2025)
- [FauForensics: Boosting Audio-Visual Deepfake Detection with Facial Action Units](https://mlsanity.com/paper/2505.08294/) (2025)
- [Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework](https://mlsanity.com/paper/2506.07358/) (2025)
- [End4: End-to-end Denoising Diffusion for Diffusion-Based Inpainting Detection](https://mlsanity.com/paper/2509.13214/) (2025)
- [Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detection](https://mlsanity.com/paper/2603.23960/) (2026)
- [Detecting AI-Generated Video: A Vision-Language Dual-View Survey](https://mlsanity.com/paper/2607.10787/) (2026)
- [Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages](https://mlsanity.com/paper/2503.21806/) (2025)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [3-Tracer: A Tri-level Temporal-Aware Framework for Audio Forgery Detection and Localization](https://mlsanity.com/paper/2511.21237/) (2025)
- [Leveraging large multimodal models for audio-video deepfake detection: a pilot study](https://mlsanity.com/paper/2602.23393/) (2026)
- [MSCT: Differential Cross-Modal Attention for Deepfake Detection](https://mlsanity.com/paper/2604.07741/) (2026)
- [Listening Deepfake Detection: A New Perspective Beyond Speaking-Centric Forgery Analysis](https://mlsanity.com/paper/2604.12650/) (2026)
- [Are DeepFakes Realistic Enough? Exploring Semantic Mismatch as a Novel Challenge](https://mlsanity.com/paper/2604.28022/) (2026)
- [DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts](https://mlsanity.com/paper/2608.23363/) (2026)
- [Cross-modal Translation via Conditional Latent Denoising for Video Deepfake Detection](https://mlsanity.com/paper/2609.33394/) (2026)
- [Physics-Aware Deepfake Detection via Distance-Speech Consistency](https://mlsanity.com/paper/2610.18485/) (2026)

## Abstract (arXiv)

Audio-visual deepfake detection scrutinizes manipulations in public video using complementary multimodal cues. Current methods, which train on fused multimodal data for multimodal targets face challenges due to uncertainties and inconsistencies in learned representations caused by independent modality manipulations in deepfake videos. To address this, we propose cross-modality and within-modality regularization to preserve modality distinctions during multimodal representation learning. Our approach includes an audio-visual transformer module for modality correspondence and a cross-modality regularization module to align paired audio-visual signals, preserving modality distinctions. Simultaneously, a within-modality regularization module refines unimodal representations with modality-specific targets to retain modal-specific details. Experimental results on the public audio-visual dataset, FakeAVCeleb, demonstrate the effectiveness and competitiveness of our approach.