# Introducing Representations of Facial Affect in Automated Multimodal Deception Detection

arXiv: https://arxiv.org/abs/2008.13369 · HTML: https://mlsanity.com/paper/2008.13369/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning](https://mlsanity.com/paper/2303.12745/) (2023)
- [MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics](https://mlsanity.com/paper/2407.12274/) (2024)
- [Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA & China](https://mlsanity.com/paper/2408.04026/) (2024)
- [Benchmarking Cross-Domain Audio-Visual Deception Detection](https://mlsanity.com/paper/2405.06995/) (2024)
- [StarLKNet: Star Mixup with Large Kernel Networks for Palm Vein Identification](https://mlsanity.com/paper/2405.12721/) (2024)
- [Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection](https://mlsanity.com/paper/2102.03673/) (2021)
- [Can lies be faked? Comparing low-stakes and high-stakes deception video datasets from a Machine Learning perspective](https://mlsanity.com/paper/2211.13035/) (2022)
- [Affect-Aware Deep Belief Network Representations for Multimodal Unsupervised Deception Detection](https://mlsanity.com/paper/2108.07897/) (2021)
- [Flexible-modal Deception Detection with Audio-Visual Adapter](https://mlsanity.com/paper/2302.05727/) (2023)
- [Expanding the Role of Affective Phenomena in Multimodal Interaction Research](https://mlsanity.com/paper/2305.10827/) (2023)
- [Dynamic Emotion and Personality Profiling for Multimodal Deception Detection](https://mlsanity.com/paper/2604.17037/) (2026)
- [LoRA-like Calibration for Multimodal Deception Detection using ATSFace Data](https://mlsanity.com/paper/2309.01383/) (2023)
- [Singing the Body Electric: The Impact of Robot Embodiment on User Expectations](https://mlsanity.com/paper/2401.06977/) (2024)

## Abstract (arXiv)

Automated deception detection systems can enhance health, justice, and security in society by helping humans detect deceivers in high-stakes situations across medical and legal domains, among others. This paper presents a novel analysis of the discriminative power of dimensional representations of facial affect for automated deception detection, along with interpretable features from visual, vocal, and verbal modalities. We used a video dataset of people communicating truthfully or deceptively in real-world, high-stakes courtroom situations. We leveraged recent advances in automated emotion recognition in-the-wild by implementing a state-of-the-art deep neural network trained on the Aff-Wild database to extract continuous representations of facial valence and facial arousal from speakers. We experimented with unimodal Support Vector Machines (SVM) and SVM-based multimodal fusion methods to identify effective features, modalities, and modeling approaches for detecting deception. Unimodal models trained on facial affect achieved an AUC of 80%, and facial affect contributed towards the highest-performing multimodal approach (adaptive boosting) that achieved an AUC of 91% when tested on speakers who were not part of training sets. This approach achieved a higher AUC than existing automated machine learning approaches that used interpretable visual, vocal, and verbal features to detect deception in this dataset, but did not use facial affect. Across all videos, deceptive and truthful speakers exhibited significant differences in facial valence and facial arousal, contributing computational support to existing psychological theories on affect and deception. The demonstrated importance of facial affect in our models informs and motivates the future development of automated, affect-aware machine learning approaches for modeling and detecting deception and other social behaviors in-the-wild.