# Multimodal Personality Recognition using Cross-Attention Transformer and Behaviour Encoding

arXiv: https://arxiv.org/abs/2112.12180 · HTML: https://mlsanity.com/paper/2112.12180/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 22 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images](https://mlsanity.com/paper/1803.00455/) (2018)
- [Multimodal End-to-End Sparse Model for Emotion Recognition](https://mlsanity.com/paper/2103.09666/) (2021)
- [Context-Aware Personality Inference in Dyadic Scenarios: Introducing the UDIVA Dataset](https://mlsanity.com/paper/2012.14259/) (2020)
- [Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition](https://mlsanity.com/paper/1609.05119/) (2016)
- [Bi-modal First Impressions Recognition using Temporally Ordered Deep Audio and Stochastic Visual Features](https://mlsanity.com/paper/1610.10048/) (2016)
- [Multimodal Video-based Apparent Personality Recognition Using Long Short-Term Memory and Convolutional Neural Networks](https://mlsanity.com/paper/1911.00381/) (2019)
- [Person Perception Biases Exposed: Revisiting the First Impressions Dataset](https://mlsanity.com/paper/2011.14906/) (2020)
- [Modern Physiognomy: An Investigation on Predicting Personality Traits and Intelligence from the Human Face](https://mlsanity.com/paper/1604.07499/) (2016)

## Built on by (most cited first, 5 of 5)

- [Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation](https://mlsanity.com/paper/2207.12817/) (2022)
- [Multimodal Vision Transformers with Forced Attention for Behavior Analysis](https://mlsanity.com/paper/2212.03968/) (2022)
- [Graph-Driven Multimodal Feature Learning Framework for Apparent Personality Assessment](https://mlsanity.com/paper/2504.11515/) (2025)
- [Loose Social-Interaction Recognition in Real-world Therapy Scenarios](https://mlsanity.com/paper/2409.20270/) (2024)
- [CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets](https://mlsanity.com/paper/2501.03332/) (2025)

## Abstract (arXiv)

Personality computing and affective computing have gained recent interest in many research areas. The datasets for the task generally have multiple modalities like video, audio, language and bio-signals. In this paper, we propose a flexible model for the task which exploits all available data. The task involves complex relations and to avoid using a large model for video processing specifically, we propose the use of behaviour encoding which boosts performance with minimal change to the model. Cross-attention using transformers has become popular in recent times and is utilised for fusion of different modalities. Since long term relations may exist, breaking the input into chunks is not desirable, thus the proposed model processes the entire input together. Our experiments show the importance of each of the above contributions