# MuMTAffect: A Multimodal Multitask Affective Framework for Personality and Emotion Recognition from Physiological Signals

arXiv: https://arxiv.org/abs/2509.04254 · HTML: https://mlsanity.com/paper/2509.04254/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2025.

## Built on by (most cited first, 4 of 4)

- [AffectAI-Capture: A Reproducible Multimodal Protocol for Small-Group Meeting Research](https://mlsanity.com/paper/2605.19794/) (2026)
- [GroupAffect-4: A Multimodal Dataset of Four-Person Collaborative Interaction](https://mlsanity.com/paper/2605.19765/) (2026)
- [RETRACE: Resilience-Guided Trait-Conditioned Craving Estimation from Wearable Physiology in Opioid Use Disorder](https://mlsanity.com/paper/2608.14947/) (2026)
- [Pseudo-Label Augmentation for Affect Sensing in Small Collaborative Groups](https://mlsanity.com/paper/2609.16077/) (2026)

## Abstract (arXiv)

We present MuMTAffect, a novel Multimodal Multitask Affective Embedding Network designed for joint emotion classification and personality prediction (re-identification) from short physiological signal segments. MuMTAffect integrates multiple physiological modalities pupil dilation, eye gaze, facial action units, and galvanic skin response using dedicated, transformer-based encoders for each modality and a fusion transformer to model cross-modal interactions. Inspired by the Theory of Constructed Emotion, the architecture explicitly separates core affect encoding (valence/arousal) from higher-level conceptualization, thereby grounding predictions in contemporary affective neuroscience. Personality trait prediction is leveraged as an auxiliary task to generate robust, user-specific affective embeddings, significantly enhancing emotion recognition performance. We evaluate MuMTAffect on the AFFEC dataset, demonstrating that stimulus-level emotional cues (Stim Emo) and galvanic skin response substantially improve arousal classification, while pupil and gaze data enhance valence discrimination. The inherent modularity of MuMTAffect allows effortless integration of additional modalities, ensuring scalability and adaptability. Extensive experiments and ablation studies underscore the efficacy of our multimodal multitask approach in creating personalized, context-aware affective computing systems, highlighting pathways for further advancements in cross-subject generalisation.