Inconsistency-aware Multimodal Schr"odinger Bridge for Deepfake Localization
Not on the map. Why?
Abstract (arXiv)
Audio-visual deepfake localization demands interval-level outputs that serve as temporal evidence. Despite recent progress, symmetric fusion under single-sided or asynchronous forgeries propagates cross-modal noise, degrading high-precision localization. We present IaMSB, an inconsistency-aware multimodal Schr"odinger Bridge (SB) that jointly estimates cross-modal consistency and performs interval-level localization. Unlike diffusion models, SB minimizes path-distribution discrepancy and yields consistency scores without explicit noise injection or denoising. With the Schr"odinger Bridge (SB), IaMSB unifies consistency estimation, cross-modal information selection, and bridge-step scheduling in one framework. Specifically, a lightweight coarse bridge first proposes candidate intervals and estimates cross-modal consistency; these statistics select cross-modal witness signals and allocate bridge steps asymmetrically across modalities. A refinement bridge then performs step-tuned fusion and outputs refined, time-aligned intervals. IaMSB anticipates single-sided and asynchronous forgeries and, using bottlenecked cross-modal interaction with step allocation, suppresses noise transfer, avoids unnecessary iterations. Across benchmarks, IaMSB stabilizes strict-IoU boundary precision, raising [email protected] by 3%~10%, and yields improved high-precision localization, particularly for single-sided forgeries.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 22 May 2026.
Builds on
31 of 41 references.
References · 31
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
- 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 594
- 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
- 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
- 2020The Creation and Detection of Deepfakes: A Surveycited by 246
- 2020Media Forensics and DeepFakes: an overviewcited by 208
- 2022Contrastive Audio-Visual Masked Autoencodercited by 142
- 2020Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localizationcited by 89
- 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
- 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
- 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
19 more
- 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
- 2022MAViL: Masked Audio-Video Learnerscited by 56
- 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
- 2023Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localizationcited by 42
- 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 40
- 2023UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localizationcited by 33
- 2024Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakescited by 32
- 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
- 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 14
- 2024Reflected Schr"odinger Bridge for Constrained Generative Modelingcited by 12
- 2024CEEBERT: Cross-Domain Inference in Early Exit BERTcited by 12
- 2024DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localizationcited by 12
- 2023Boundary-Denoising for Video Activity Localizationcited by 10
- 2024Coarse-to-Fine Proposal Refinement Framework for Audio Temporal Forgery Detection and Localizationcited by 9
- 2024Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localizationcited by 6
- 2023Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video Parsingcited by 5
- 2024Faster Diffusion Action Segmentationcited by 2
- 2025Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videoscited by 2
- 2025Generic Event Boundary Detection via Denoising Diffusioncited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Geometry-based Schr"odinger Bridges for Trustworthy Multimodal Fusion
- 2025Multi-modal Deepfake Detection and Localization with FPN-Transformer
- 2026Audio-Anchored Fusion of Multi-Ratio DiT Reconstruction Residuals for Cross-Domain Audio Deepfake Detection
- 2025Schr"odinger Bridge Consistency Trajectory Models for Speech Enhancement
- 2026X-AVDT: Audio-Visual Cross-Attention for Robust Deepfake Detection
- 2026Rectified Schr"odinger Bridge Matching for Few-Step Visual Navigation
- 2026ToolDF: Tool-Integrated Reasoning for Mixed-Authenticity Audio Deepfake Detection
- 2026MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection