# Looking At The Body: Automatic Analysis of Body Gestures and Self-Adaptors in Psychological Distress

arXiv: https://arxiv.org/abs/2007.15815 · HTML: https://mlsanity.com/paper/2007.15815/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2020.

## Built on by (most cited first, 4 of 4)

- [Promoting the Responsible Development of Speech Datasets for Mental Health and Neurological Disorders Research](https://mlsanity.com/paper/2406.04116/) (2024)
- [A Comprehensive Review of Datasets for Clinical Mental Health AI Systems](https://mlsanity.com/paper/2508.09809/) (2025)
- [Automatic Depression Assessment using Machine Learning: A Comprehensive Survey](https://mlsanity.com/paper/2506.18915/) (2025)
- [A Multimodal Neural Network for Recognizing Subjective Self-Disclosure Towards Social Robots](https://mlsanity.com/paper/2508.10828/) (2025)

## Abstract (arXiv)

Psychological distress is a significant and growing issue in society. Automatic detection, assessment, and analysis of such distress is an active area of research. Compared to modalities such as face, head, and vocal, research investigating the use of the body modality for these tasks is relatively sparse. This is, in part, due to the limited available datasets and difficulty in automatically extracting useful body features. Recent advances in pose estimation and deep learning have enabled new approaches to this modality and domain. To enable this research, we have collected and analyzed a new dataset containing full body videos for short interviews and self-reported distress labels. We propose a novel method to automatically detect self-adaptors and fidgeting, a subset of self-adaptors that has been shown to be correlated with psychological distress. We perform analysis on statistical body gestures and fidgeting features to explore how distress levels affect participants' behaviors. We then propose a multi-modal approach that combines different feature representations using Multi-modal Deep Denoising Auto-Encoders and Improved Fisher Vector Encoding. We demonstrate that our proposed model, combining audio-visual features with automatically detected fidgeting behavioral cues, can successfully predict distress levels in a dataset labeled with self-reported anxiety and depression levels.