# Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches

arXiv: https://arxiv.org/abs/2307.06625 · HTML: https://mlsanity.com/paper/2307.06625/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 83 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [RetinaFace: Single-stage Dense Face Localisation in the Wild](https://mlsanity.com/paper/1905.00641/) (2019)
- [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1711.09017/) (2017)
- [A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation](https://mlsanity.com/paper/2001.00187/) (2020)
- [L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments](https://mlsanity.com/paper/2203.03339/) (2022)
- [Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms](https://mlsanity.com/paper/2201.12728/) (2022)
- [WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose](https://mlsanity.com/paper/2005.10353/) (2020)
- [Appearance-Based Gaze Estimation Using Dilated-Convolutions](https://mlsanity.com/paper/1903.07296/) (2019)
- [A Vector-based Representation to Enhance Head Pose Estimation](https://mlsanity.com/paper/2010.07184/) (2020)
- [Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution](https://mlsanity.com/paper/2004.09681/) (2020)
- [Deception Detection in Videos](https://mlsanity.com/paper/1712.04415/) (2017)
- [Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?](https://mlsanity.com/paper/1510.02969/) (2015)
- [PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship](https://mlsanity.com/paper/1811.08657/) (2018)
- [Deception Detection in Group Video Conversations using Dynamic Interaction Networks](https://mlsanity.com/paper/2106.06163/) (2021)
- [The politics of deceptive borders: 'biomarkers of deceit' and the case of iBorderCtrl](https://mlsanity.com/paper/1911.09156/) (2019)

## Abstract (arXiv)

Deception detection is an interdisciplinary field attracting researchers from psychology, criminology, computer science, and economics. We propose a multimodal approach combining deep learning and discriminative models for automated deception detection. Using video modalities, we employ convolutional end-to-end learning to analyze gaze, head pose, and facial expressions, achieving promising results compared to state-of-the-art methods. Due to limited training data, we also utilize discriminative models for deception detection. Although sequence-to-class approaches are explored, discriminative models outperform them due to data scarcity. Our approach is evaluated on five datasets, including a new Rolling-Dice Experiment motivated by economic factors. Results indicate that facial expressions outperform gaze and head pose, and combining modalities with feature selection enhances detection performance. Differences in expressed features across datasets emphasize the importance of scenario-specific training data and the influence of context on deceptive behavior. Cross-dataset experiments reinforce these findings. Despite the challenges posed by low-stake datasets, including the Rolling-Dice Experiment, deception detection performance exceeds chance levels. Our proposed multimodal approach and comprehensive evaluation shed light on the potential of automating deception detection from video modalities, opening avenues for future research.