# Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning

arXiv: https://arxiv.org/abs/2303.12745 · HTML: https://mlsanity.com/paper/2303.12745/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 55 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Convolutional Bypasses Are Better Vision Transformer Adapters](https://mlsanity.com/paper/2207.07039/) (2022)
- [Fine-tuning wav2vec2 for speaker recognition](https://mlsanity.com/paper/2109.15053/) (2021)
- [Deception Detection in Videos](https://mlsanity.com/paper/1712.04415/) (2017)
- [Attention-Augmented End-to-End Multi-Task Learning for Emotion Prediction from Speech](https://mlsanity.com/paper/1903.12424/) (2019)
- [Deception Detection and Remote Physiological Monitoring: A Dataset and Baseline Experimental Results](https://mlsanity.com/paper/2106.06583/) (2021)
- [A Deep Learning Approach for Multimodal Deception Detection](https://mlsanity.com/paper/1803.00344/) (2018)
- [Face-Focused Cross-Stream Network for Deception Detection in Videos](https://mlsanity.com/paper/1812.04429/) (2018)
- [Introducing Representations of Facial Affect in Automated Multimodal Deception Detection](https://mlsanity.com/paper/2008.13369/) (2020)

## Built on by (most cited first, 14 of 14)

- [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024)
- [Benchmarking Cross-Domain Audio-Visual Deception Detection](https://mlsanity.com/paper/2405.06995/) (2024)
- [SVC 2025: the First Multimodal Deception Detection Challenge](https://mlsanity.com/paper/2508.04129/) (2025)
- [Text-guided Fine-Grained Video Anomaly Understanding](https://mlsanity.com/paper/2511.00524/) (2025)
- [POLygraph: Polish Fake News Dataset](https://mlsanity.com/paper/2407.01393/) (2024)
- [AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2508.07608/) (2025)
- [DeceptionX: From Multimodal Evidence to Explainable Deception Detection](https://mlsanity.com/paper/2606.11385/) (2026)
- [Enhancing Lie Detection Accuracy: A Comparative Study of Classic ML, CNN, and GCN Models using Audio-Visual Features](https://mlsanity.com/paper/2411.08885/) (2024)
- [Multi-source Multimodal Progressive Domain Adaption for Audio-Visual Deception Detection](https://mlsanity.com/paper/2508.12842/) (2025)
- [DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning](https://mlsanity.com/paper/2603.23916/) (2026)
- [MuDD: A Multimodal Deception Detection Dataset and GSR-Guided Progressive Distillation for Non-Contact Deception Detection](https://mlsanity.com/paper/2603.26064/) (2026)
- [SVC 2026: the Second Multimodal Deception Detection Challenge and the First Domain Generalized Remote Physiological Measurement Challenge](https://mlsanity.com/paper/2604.05748/) (2026)
- [ThinkDeception: A Progressive Reinforcement Learning Framework for Interpretable Multimodal Deception Detection](https://mlsanity.com/paper/2606.18988/) (2026)
- [Catching Lies Without Sending the Video: Privacy-Preserving Multimodal Deception Detection](https://mlsanity.com/paper/2606.22699/) (2026)

## Abstract (arXiv)

Deception detection in conversations is a challenging yet important task, having pivotal applications in many fields such as credibility assessment in business, multimedia anti-frauds, and custom security. Despite this, deception detection research is hindered by the lack of high-quality deception datasets, as well as the difficulties of learning multimodal features effectively. To address this issue, we introduce DOLOSfootnote The name ``DOLOS" comes from Greek mythology., the largest gameshow deception detection dataset with rich deceptive conversations. DOLOS includes 1,675 video clips featuring 213 subjects, and it has been labeled with audio-visual feature annotations. We provide train-test, duration, and gender protocols to investigate the impact of different factors. We benchmark our dataset on previously proposed deception detection approaches. To further improve the performance by fine-tuning fewer parameters, we propose Parameter-Efficient Crossmodal Learning (PECL), where a Uniform Temporal Adapter (UT-Adapter) explores temporal attention in transformer-based architectures, and a crossmodal fusion module, Plug-in Audio-Visual Fusion (PAVF), combines crossmodal information from audio-visual features. Based on the rich fine-grained audio-visual annotations on DOLOS, we also exploit multi-task learning to enhance performance by concurrently predicting deception and audio-visual features. Experimental results demonstrate the desired quality of the DOLOS dataset and the effectiveness of the PECL. The DOLOS dataset and the source codes are available at https://github.com/NMS05/Audio-Visual-Deception-Detection-DOLOS-Dataset-and-Parameter-Efficient-Crossmodal-Learning/tree/main.