# Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Prediction

arXiv: https://arxiv.org/abs/1805.00705 · HTML: https://mlsanity.com/paper/1805.00705/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2018.

## Built on by (most cited first, 22 of 22)

- [Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations](https://mlsanity.com/paper/2210.17444/) (2022)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [The Effects of Embodiment and Personality Expression on Learning in LLM-based Educational Agents](https://mlsanity.com/paper/2407.10993/) (2024)
- [Hybrid Multimodal Fusion for Humor Detection](https://mlsanity.com/paper/2209.11949/) (2022)
- [Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space](https://mlsanity.com/paper/2404.08923/) (2024)
- [Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction](https://mlsanity.com/paper/2301.02383/) (2023)
- [ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization](https://mlsanity.com/paper/2208.09414/) (2022)
- [On Text-based Personality Computing: Challenges and Future Directions](https://mlsanity.com/paper/2212.06711/) (2022)
- [Inferred vs traditional personality assessment: are we predicting the same thing?](https://mlsanity.com/paper/2103.09632/) (2021)
- [Explainable Human-centered Traits from Head Motion and Facial Expression Dynamics](https://mlsanity.com/paper/2302.09817/) (2023)
- [Personality-aware Human-centric Multimodal Reasoning: A New Task, Dataset and Baselines](https://mlsanity.com/paper/2304.02313/) (2023)
- [Extroversion or Introversion? Controlling The Personality of Your Large Language Models](https://mlsanity.com/paper/2406.04583/) (2024)
- [Twenty Years of Personality Computing: Threats, Challenges and Future Directions](https://mlsanity.com/paper/2503.02082/) (2025)
- [Curriculum Learning Meets Weakly Supervised Modality Correlation Learning](https://mlsanity.com/paper/2212.07619/) (2022)
- [Multi-Modality Collaborative Learning for Sentiment Analysis](https://mlsanity.com/paper/2501.12424/) (2025)
- [Personality Editing for Language Models through Adjusting Self-Referential Queries](https://mlsanity.com/paper/2502.11789/) (2025)
- [Personality Style Recognition via Machine Learning: Identifying Anaclitic and Introjective Personality Styles from Patients' Speech](https://mlsanity.com/paper/2311.04088/) (2023)
- [Enhancing Apparent Personality Trait Analysis with Cross-Modal Embeddings](https://mlsanity.com/paper/2405.03846/) (2024)
- [Multi-modal expressive personality recognition in data non-ideal audiovisual based on multi-scale feature enhancement and modal augment](https://mlsanity.com/paper/2503.06108/) (2025)
- [Joint Modeling of Big Five and HEXACO for Multimodal Apparent Personality-trait Recognition](https://mlsanity.com/paper/2510.14203/) (2025)

## Abstract (arXiv)

We propose a tri-modal architecture to predict Big Five personality trait scores from video clips with different channels for audio, text, and video data. For each channel, stacked Convolutional Neural Networks are employed. The channels are fused both on decision-level and by concatenating their respective fully connected layers. It is shown that a multimodal fusion approach outperforms each single modality channel, with an improvement of 9.4% over the best individual modality (video). Full backpropagation is also shown to be better than a linear combination of modalities, meaning complex interactions between modalities can be leveraged to build better models. Furthermore, we can see the prediction relevance of each modality for each trait. The described model can be used to increase the emotional intelligence of virtual agents.