# Benchmarking Cross-Domain Audio-Visual Deception Detection

arXiv: https://arxiv.org/abs/2405.06995 · HTML: https://mlsanity.com/paper/2405.06995/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 11 May 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2104.09937/) (2021)
- [Domain Generalization via Shuffled Style Assembly for Face Anti-Spoofing](https://mlsanity.com/paper/2203.05340/) (2022)
- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Multi-View Spatial-Temporal Graph Convolutional Networks with Domain Generalization for Sleep Stage Classification](https://mlsanity.com/paper/2109.01824/) (2021)
- [Towards Domain Generalization for Multi-view 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2303.01686/) (2023)
- [Deception Detection in Videos](https://mlsanity.com/paper/1712.04415/) (2017)
- [Domain Generalization via Balancing Training Difficulty and Model Capability](https://mlsanity.com/paper/2309.00844/) (2023)
- [Face-Focused Cross-Stream Network for Deception Detection in Videos](https://mlsanity.com/paper/1812.04429/) (2018)
- [Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning](https://mlsanity.com/paper/2303.12745/) (2023)
- [Introducing Representations of Facial Affect in Automated Multimodal Deception Detection](https://mlsanity.com/paper/2008.13369/) (2020)
- [Data Leakage and Evaluation Issues in Micro-Expression Analysis](https://mlsanity.com/paper/2211.11425/) (2022)
- [Deception Detection in Group Video Conversations using Dynamic Interaction Networks](https://mlsanity.com/paper/2106.06163/) (2021)
- [Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection](https://mlsanity.com/paper/2102.03673/) (2021)
- [Deception Detection in Videos using the Facial Action Coding System](https://mlsanity.com/paper/2105.13659/) (2021)

## Built on by (most cited first, 6 of 6)

- [SVC 2025: the First Multimodal Deception Detection Challenge](https://mlsanity.com/paper/2508.04129/) (2025)
- [Text-guided Fine-Grained Video Anomaly Understanding](https://mlsanity.com/paper/2511.00524/) (2025)
- [AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2508.07608/) (2025)
- [Multi-source Multimodal Progressive Domain Adaption for Audio-Visual Deception Detection](https://mlsanity.com/paper/2508.12842/) (2025)
- [DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning](https://mlsanity.com/paper/2603.23916/) (2026)
- [ThinkDeception: A Progressive Reinforcement Learning Framework for Interpretable Multimodal Deception Detection](https://mlsanity.com/paper/2606.18988/) (2026)

## Abstract (arXiv)

Automated deception detection is crucial for assisting humans in accurately assessing truthfulness and identifying deceptive behavior. Conventional contact-based techniques, like polygraph devices, rely on physiological signals to determine the authenticity of an individual's statements. Nevertheless, recent developments in automated deception detection have demonstrated that multimodal features derived from both audio and video modalities may outperform human observers on publicly available datasets. Despite these positive findings, the generalizability of existing audio-visual deception detection approaches across different scenarios remains largely unexplored. To close this gap, we present the first cross-domain audio-visual deception detection benchmark, that enables us to assess how well these methods generalize for use in real-world scenarios. We used widely adopted audio and visual features and different architectures for benchmarking, comparing single-to-single and multi-to-single domain generalization performance. To further exploit the impacts using data from multiple source domains for training, we investigate three types of domain sampling strategies, including domain-simultaneous, domain-alternating, and domain-by-domain for multi-to-single domain generalization evaluation. We also propose an algorithm to enhance the generalization performance by maximizing the gradient inner products between modality encoders, named ``MM-IDGM". Furthermore, we proposed the Attention-Mixer fusion method to improve performance, and we believe that this new cross-domain benchmark will facilitate future research in audio-visual deception detection.