# Enhancing Apparent Personality Trait Analysis with Cross-Modal Embeddings

arXiv: https://arxiv.org/abs/2405.03846 · HTML: https://mlsanity.com/paper/2405.03846/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 12 of 12)

- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems](https://mlsanity.com/paper/1512.08756/) (2015)
- [Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Prediction](https://mlsanity.com/paper/1805.00705/) (2018)
- [PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship](https://mlsanity.com/paper/1811.08657/) (2018)
- [First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis](https://mlsanity.com/paper/1804.08046/) (2018)
- [Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos](https://mlsanity.com/paper/1802.00745/) (2018)
- [EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings](https://mlsanity.com/paper/1907.10428/) (2019)

## Abstract (arXiv)

Automatic personality trait assessment is essential for high-quality human-machine interactions. Systems capable of human behavior analysis could be used for self-driving cars, medical research, and surveillance, among many others. We present a multimodal deep neural network with a Siamese extension for apparent personality trait prediction trained on short video recordings and exploiting modality invariant embeddings. Acoustic, visual, and textual information are utilized to reach high-performance solutions in this task. Due to the highly centralized target distribution of the analyzed dataset, the changes in the third digit are relevant. Our proposed method addresses the challenge of under-represented extreme values, achieves 0.0033 MAE average improvement, and shows a clear advantage over the baseline multimodal DNN without the introduced module.