# A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

arXiv: https://arxiv.org/abs/2406.14176 · HTML: https://mlsanity.com/paper/2406.14176/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization](https://mlsanity.com/paper/2005.14405/) (2020)
- [StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator](https://mlsanity.com/paper/2305.05445/) (2023)
- [Spatiotemporal Inconsistency Learning for DeepFake Video Detection](https://mlsanity.com/paper/2109.01860/) (2021)
- [One-class Learning Towards Synthetic Voice Spoofing Detection](https://mlsanity.com/paper/2010.13995/) (2020)
- [Perfect match: Improved cross-modal embeddings for audio-visual synchronisation](https://mlsanity.com/paper/1809.08001/) (2018)
- [Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors](https://mlsanity.com/paper/2109.02993/) (2021)
- [Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection](https://mlsanity.com/paper/2401.05746/) (2024)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [SAMO: Speaker Attractor Multi-Center One-Class Learning for Voice Anti-Spoofing](https://mlsanity.com/paper/2211.02718/) (2022)
- [One-Class Knowledge Distillation for Spoofing Speech Detection](https://mlsanity.com/paper/2309.08285/) (2023)
- [Rethinking Audio-visual Synchronization for Active Speaker Detection](https://mlsanity.com/paper/2206.10421/) (2022)
- [Integrating Audio-Visual Features for Multimodal Deepfake Detection](https://mlsanity.com/paper/2310.03827/) (2023)
- [AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection](https://mlsanity.com/paper/2310.13103/) (2023)

## Built on by (most cited first, 3 of 3)

- [Investigating self-supervised representations for audio-visual deepfake detection](https://mlsanity.com/paper/2511.17181/) (2025)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [GEM-TFL: Bridging Weak and Full Supervision for Forgery Localization through EM-Guided Decomposition and Temporal Refinement](https://mlsanity.com/paper/2603.05095/) (2026)

## Abstract (arXiv)

This paper addresses the challenge of developing a robust audio-visual deepfake detection model. In practical use cases, new generation algorithms are continually emerging, and these algorithms are not encountered during the development of detection methods. This calls for the generalization ability of the method. Additionally, to ensure the credibility of detection methods, it is beneficial for the model to interpret which cues from the video indicate it is fake. Motivated by these considerations, we then propose a multi-stream fusion approach with one-class learning as a representation-level regularization technique. We study the generalization problem of audio-visual deepfake detection by creating a new benchmark by extending and re-splitting the existing FakeAVCeleb dataset. The benchmark contains four categories of fake videos (Real Audio-Fake Visual, Fake Audio-Fake Visual, Fake Audio-Real Visual, and Unsynchronized videos). The experimental results demonstrate that our approach surpasses the previous models by a large margin. Furthermore, our proposed framework offers interpretability, indicating which modality the model identifies as more likely to be fake. The source code is released at https://github.com/bok-bok/MSOC.