# Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition

arXiv: https://arxiv.org/abs/2110.13570 · HTML: https://mlsanity.com/paper/2110.13570/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 105 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Residual Gated Graph ConvNets](https://mlsanity.com/paper/1711.07553/) (2017)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Neural Architecture Optimization](https://mlsanity.com/paper/1808.07233/) (2018)
- [Genetic CNN](https://mlsanity.com/paper/1703.01513/) (2017)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures](https://mlsanity.com/paper/1806.08198/) (2018)
- [EdgeNets:Edge Varying Graph Neural Networks](https://mlsanity.com/paper/2001.07620/) (2020)
- [Context-Aware Personality Inference in Dyadic Scenarios: Introducing the UDIVA Dataset](https://mlsanity.com/paper/2012.14259/) (2020)
- [Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactions](https://mlsanity.com/paper/2109.09487/) (2021)
- [Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition](https://mlsanity.com/paper/1609.05119/) (2016)
- [Generating Talking Face Landmarks from Speech](https://mlsanity.com/paper/1803.09803/) (2018)
- [PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship](https://mlsanity.com/paper/1811.08657/) (2018)
- [Bi-modal First Impressions Recognition using Temporally Ordered Deep Audio and Stochastic Visual Features](https://mlsanity.com/paper/1610.10048/) (2016)
- [GraphTCN: Spatio-Temporal Interaction Modeling for Human Trajectory Prediction](https://mlsanity.com/paper/2003.07167/) (2020)
- [On the Effect of Observed Subject Biases in Apparent Personality Analysis from Audio-visual Signals](https://mlsanity.com/paper/1909.05568/) (2019)

## Built on by (most cited first, 5 of 5)

- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [An Open-source Benchmark of Deep Learning Models for Audio-visual Apparent and Self-reported Personality Recognition](https://mlsanity.com/paper/2210.09138/) (2022)
- [GRATIS: Deep Learning Graph Representation with Task-specific Topology and Multi-dimensional Edge Features](https://mlsanity.com/paper/2211.12482/) (2022)
- [Domain-specific Learning of Multi-scale Facial Dynamics for Apparent Personality Traits Prediction](https://mlsanity.com/paper/2209.04148/) (2022)
- [Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding](https://mlsanity.com/paper/2510.10682/) (2025)

## Abstract (arXiv)

This approach builds on two following findings in cognitive science: (i) human cognition partially determines expressed behaviour and is directly linked to true personality traits; and (ii) in dyadic interactions individuals' nonverbal behaviours are influenced by their conversational partner behaviours. In this context, we hypothesise that during a dyadic interaction, a target subject's facial reactions are driven by two main factors, i.e. their internal (person-specific) cognitive process, and the externalised nonverbal behaviours of their conversational partner. Consequently, we propose to represent the target subjects (defined as the listener) person-specific cognition in the form of a person-specific CNN architecture that has unique architectural parameters and depth, which takes audio-visual non-verbal cues displayed by the conversational partner (defined as the speaker) as input, and is able to reproduce the target subject's facial reactions. Each person-specific CNN is explored by the Neural Architecture Search (NAS) and a novel adaptive loss function, which is then represented as a graph representation for recognising the target subject's true personality. Experimental results not only show that the produced graph representations are well associated with target subjects' personality traits in both human-human and human-machine interaction scenarios, and outperform the existing approaches with significant advantages, but also demonstrate that the proposed novel strategies such as adaptive loss, and the end-to-end vertices/edges feature learning, help the proposed approach in learning more reliable personality representations.