# Decoding Psychological States Through Movement: Inferring Human Kinesic Functions with Application to Built Environments

arXiv: https://arxiv.org/abs/2601.17194 · HTML: https://mlsanity.com/paper/2601.17194/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 87 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Builds on (most cited first, 10 of 10)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Socially and Contextually Aware Human Motion and Pose Forecasting](https://mlsanity.com/paper/2007.06843/) (2020)
- [V4D:4D Convolutional Neural Networks for Video-level Representation Learning](https://mlsanity.com/paper/2002.07442/) (2020)
- [Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground](https://mlsanity.com/paper/2211.12883/) (2022)
- [To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations](https://mlsanity.com/paper/1910.02181/) (2019)
- [AIR-Act2Act: Human-human interaction dataset for teaching non-verbal social behaviors to robots](https://mlsanity.com/paper/2009.02041/) (2020)
- [The architecture of innovation: Tracking face-to-face interactions with ubicomp technologies](https://mlsanity.com/paper/1406.6829/) (2014)
- [Read the Room: Inferring Social Context Through Dyadic Interaction Recognition in Cyber-physical-social Infrastructure Systems](https://mlsanity.com/paper/2510.04854/) (2025)

## Abstract (arXiv)

Social infrastructure and other built environments are increasingly expected to support well-being and community resilience by enabling social interaction. Yet in civil and built-environment research, there is no consistent and privacy-preserving way to represent and measure socially meaningful interaction in these spaces, leaving studies to operationalize "interaction" differently across contexts and limiting practitioners' ability to evaluate whether design interventions are changing the forms of interaction that social capital theory predicts should matter. To address this field-level and methodological gap, we introduce the Dyadic User Engagement DataseT (DUET) dataset and an embedded kinesics recognition framework that operationalize Ekman and Friesen's kinesics taxonomy as a function-level interaction vocabulary aligned with social capital-relevant behaviors (e.g., reciprocity and attention coordination). DUET captures 12 dyadic interactions spanning all five kinesic functions-emblems, illustrators, affect displays, adaptors, and regulators-across four sensing modalities and three built-environment contexts, enabling privacy-preserving analysis of communicative intent through movement. Benchmarking six open-source, state-of-the-art human activity recognition models quantifies the difficulty of communicative-function recognition on DUET and highlights the limitations of ubiquitous monadic, action-level recognition when extended to dyadic, socially grounded interaction measurement. Building on DUET, our recognition framework infers communicative function directly from privacy-preserving skeletal motion without handcrafted action-to-function dictionaries; using a transfer-learning architecture, it reveals structured clustering of kinesic functions and a strong association between representation quality and classification performance while generalizing across subjects and contexts.